Oct 31 01:19:30.055384 ip-10-0-130-49 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:19:30.480021 ip-10-0-130-49 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:19:30.480021 ip-10-0-130-49 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:19:30.480021 ip-10-0-130-49 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:19:30.480021 ip-10-0-130-49 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:19:30.487078 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.486972    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:30.487078 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487075    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:30.487078 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487079    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:30.487078 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487082    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:30.487078 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487085    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487088    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487091    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487095    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487097    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487100    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487102    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487105    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487108    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487111    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487113    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487116    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487118    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487121    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487124    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487126    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487129    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487131    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487134    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:30.487267 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487137    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487139    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487142    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487145    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487147    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487150    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487152    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487155    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487158    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487162    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487164    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487167    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487169    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487172    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487175    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487178    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487181    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487184    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487186    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:30.487702 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487189    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487193    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487195    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487198    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487200    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487203    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487205    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487208    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487212    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487214    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487217    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487220    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487222    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487225    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487227    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487230    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487232    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487235    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487237    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487240    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:30.488185 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487242    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487245    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487248    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487250    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487253    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487255    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487258    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487261    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487264    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487267    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487270    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487273    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487276    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487279    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487282    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487284    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487286    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487289    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487291    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487294    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:30.488742 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487296    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487301    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487305    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487308    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487311    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487313    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487316    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487318    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487321    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487324    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487326    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487329    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487331    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487334    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487336    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487339    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487341    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487344    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487346    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:30.489254 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487349    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487352    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487355    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487360    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.487364    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488154    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488162    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488166    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488168    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488171    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488174    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488177    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488179    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488182    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488184    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488188    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488191    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488193    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488196    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:30.489682 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488199    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488201    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488204    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488206    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488209    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488211    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488214    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488216    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488219    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488221    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488224    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488227    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488229    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488232    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488236    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488238    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488241    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488244    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488246    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488249    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:30.490144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488252    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488254    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488257    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488259    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488262    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488264    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488267    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488269    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488272    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488276    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488278    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488281    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488283    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488286    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488288    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488291    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488293    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488296    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488298    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:30.490629 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488301    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488303    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488306    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488308    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488310    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488313    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488316    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488319    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488322    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488325    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488327    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488330    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488332    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488335    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488337    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488340    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488343    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488346    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488348    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488351    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:30.491092 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488353    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488356    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488358    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488362    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488364    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488367    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488369    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488372    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488375    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488377    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488380    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488382    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488385    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488388    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488391    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488395    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488399    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488403    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488406    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:30.491564 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488409    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488412    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488416    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488418    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488421    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488423    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488426    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488428    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488431    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488433    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488436    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488438    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488441    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.488443    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489019    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489025    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489033    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489055    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489058    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489061    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:30.492012 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489064    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489067    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489069    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489072    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489075    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489077    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489080    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489083    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489085    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489088    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489090    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489094    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489096    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489099    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489102    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489105    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489107    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489109    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489112    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489114    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:30.492538 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489117    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489119    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489121    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489124    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489126    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489129    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489131    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489134    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489137    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489140    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489143    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489145    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489147    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489150    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489153    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489155    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489158    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489161    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489163    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:30.493005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489166    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489168    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489171    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489173    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489176    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489179    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489182    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489184    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489187    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489189    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489192    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489195    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489197    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489200    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489202    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489205    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489207    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489210    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489212    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489215    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:30.493480 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489218    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489220    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489223    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489226    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489228    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489233    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489237    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489240    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489243    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489245    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489248    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489252    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489255    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489257    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489260    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489263    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489266    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489269    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489272    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:30.493979 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489274    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489277    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489280    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489282    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489285    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489287    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489290    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489292    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489295    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489298    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489300    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489303    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489305    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489308    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489310    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489313    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489315    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489319    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489323    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:30.494445 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489327    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489330    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.489332    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491082    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491092    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491099    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491104    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491108    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491111    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491116    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491121    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491124    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491127    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491131    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491134    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491137    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491140    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491143    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491146    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491148    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491152    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491156    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491159    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491162    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:19:30.494903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491165    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491168    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491178    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491181    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491184    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491187    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491189    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491193    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491196    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491199    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491202    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491207    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491210    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491213    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491215    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491218    2574 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491221    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491225    2574 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491229    2574 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491232    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491235    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491238    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491242    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491245    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491248    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:19:30.495503 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491251    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491254    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491257    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491260    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491263    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491266    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491268    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491271    2574 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491275    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491278    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491282    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491285    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491288    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491291    2574 flags.go:64] FLAG: --help="false"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491294    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491298    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491301    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491304    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491310    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491313    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491316    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491319    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491322    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:19:30.496142 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491325    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491328    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491331    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491334    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491337    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491340    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491343    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491346    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491348    2574 flags.go:64] FLAG: --lock-file=""
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491351    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491354    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491357    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491362    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491365    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491369    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491372    2574 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491374    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491378    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491380    2574 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491383    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491387    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491390    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491394    2574 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491397    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491400    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:19:30.496697 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491404    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491407    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491411    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491414    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491417    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491424    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491427    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491430    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491433    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491436    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491439    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491442    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491444    2574 flags.go:64] FLAG: --port="10250"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491448    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491451    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0515dd301b49ade2b"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491454    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491457    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491460    2574 flags.go:64] FLAG: --register-node="true"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491463    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491467    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491470    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491472    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491475    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491479    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491482    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491485    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:19:30.497355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491488    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491491    2574 flags.go:64] FLAG: --runonce="false"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491494    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491497    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491500    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491503    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491505    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491509    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491514    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491517    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491520    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491522    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491525    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491528    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491533    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491536    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491539    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491544    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491547    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491549    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491553    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491556    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491559    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491562    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491564    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:19:30.497956 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491568    2574 flags.go:64] FLAG: --v="2"
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491572    2574 flags.go:64] FLAG: --version="false"
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491576    2574 flags.go:64] FLAG: --vmodule=""
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491581    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.491584    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491721    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491725    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491728    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491731    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491733    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491736    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491738    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491741    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491743    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491746    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491750    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491753    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491755    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491758    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491760    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491763    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:30.498900 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491767    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491770    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491772    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491775    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491777    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491780    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491782    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491785    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491787    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491790    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491792    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491795    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491797    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491800    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491802    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491805    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491807    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491810    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491812    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:30.499809 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491815    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491817    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491820    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491822    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491825    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491827    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491830    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491833    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491837    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491839    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491842    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491845    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491847    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491850    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491853    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491856    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491858    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491861    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491864    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491866    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:30.500622 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491869    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491871    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491874    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491876    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491879    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491881    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491884    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491887    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491889    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491892    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491894    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491897    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491899    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491902    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491904    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491907    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491909    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491911    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491914    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:30.501498 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491917    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491919    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491923    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491926    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491928    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491931    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491936    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491940    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491943    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491946    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491949    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491951    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491954    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491956    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491959    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491963    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491966    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491970    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491973    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:30.502437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491976    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491978    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491981    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491983    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491986    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491988    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491991    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491994    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491996    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.491999    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492001    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492004    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492007    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.492032    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492103    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492107    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:30.503128 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492113    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492116    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492119    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492122    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492126    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492129    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492131    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492134    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492136    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492139    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492141    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492144    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492146    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492149    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492151    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492154    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492156    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492159    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492161    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:30.503518 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492164    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492167    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492169    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492172    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492174    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492177    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492179    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492182    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492184    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492188    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492191    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492195    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492199    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492202    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492205    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492209    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492211    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492215    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492218    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:30.504294 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492221    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492224    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492228    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492231    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492234    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492237    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492239    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492242    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492244    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492247    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492249    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492252    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492255    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492258    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492260    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492263    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492265    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492268    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492271    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492273    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:30.505082 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492276    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492278    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492282    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492285    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492287    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492289    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492292    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492294    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492298    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492301    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492305    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492308    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492310    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492313    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492315    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492318    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492320    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492323    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492325    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492328    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:30.505574 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492330    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492333    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492335    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492337    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492340    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492343    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492345    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492348    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492350    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492353    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492356    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492358    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492360    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492363    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492367    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492369    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492372    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492374    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492377    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:30.506256 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492380    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492383    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492387    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492390    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492393    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492396    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.492398    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.500732    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.500757    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.500780    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.500789    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500875    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500883    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500889    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500895    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500899    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500904    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500908    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500913    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500917    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:30.506709 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500922    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500926    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500930    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500935    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500939    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500943    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500948    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500953    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500957    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500961    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500966    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500970    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500974    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500978    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500982    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500987    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500991    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.500996    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501000    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501005    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:30.507304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501011    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501015    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501019    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501023    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501028    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501032    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501055    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501060    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501065    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501069    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501073    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501077    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501081    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501086    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501090    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501093    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501098    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501102    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501106    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501111    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:30.507762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501115    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501119    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501123    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501127    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501132    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501136    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501141    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501145    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501150    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501154    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501158    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501162    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501167    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501172    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501177    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501182    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501186    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501190    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501194    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:30.508326 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501198    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501203    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501207    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501211    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501215    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501219    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501224    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501231    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501237    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501241    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501245    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501249    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501253    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501258    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501262    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501266    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501270    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501274    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501279    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:30.508801 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501283    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501287    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501291    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501295    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501299    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501304    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501308    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501312    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501318    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501322    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501326    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501330    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501334    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501338    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501343    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501347    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501353    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501359    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:30.509387 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501363    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.501403    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501617    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501626    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501630    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501635    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501639    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501643    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501647    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501651    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501656    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501660    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501665    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501669    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501673    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501677    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:30.510060 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501681    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501686    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501690    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501695    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501699    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501704    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501709    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501716    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501721    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501725    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501729    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501734    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501738    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501742    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501747    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501751    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501755    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501760    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:30.510451 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501767    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501772    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501776    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501781    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501785    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501789    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501793    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501798    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501802    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501806    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501811    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501815    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501819    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501824    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501829    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501833    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501837    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501841    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501846    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:30.511030 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501850    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501855    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501859    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501864    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501868    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501873    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501877    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501881    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501885    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501889    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501893    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501897    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501902    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501906    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501910    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501915    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501919    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501923    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501927    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501931    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:30.511762 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501936    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501940    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501944    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501948    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501952    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501958    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501962    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501966    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501971    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501975    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501980    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501984    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501989    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501993    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.501997    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502002    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502007    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502011    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502015    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502019    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:30.512384 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502023    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502027    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502032    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502054    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502058    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502062    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502066    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502069    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502077    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502080    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502084    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502089    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502092    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502096    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502100    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.502139    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:19:30.512938 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502215    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502223    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502229    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502233    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502237    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502242    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502246    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502250    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502255    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502259    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502264    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502268    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502272    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502278    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502282    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502286    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502291    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502295    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502299    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502304    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:30.513341 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502308    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502312    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502319    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502325    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502333    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502338    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502343    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502347    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502352    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502356    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502360    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502365    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502371    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502377    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502381    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502386    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502399    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502404    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:30.513824 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502408    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502413    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502417    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502421    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502426    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502430    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502434    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502438    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502444    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502448    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502452    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502456    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502460    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502464    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502468    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502472    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502476    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502481    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502489    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:30.514287 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502493    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502498    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502502    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502506    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502511    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502515    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502520    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502524    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502528    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502532    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502536    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502541    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502545    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502549    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502553    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502559    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502564    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502568    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502573    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502577    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:30.514799 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502582    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502586    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502591    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502595    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502599    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502604    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502608    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502612    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502617    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502621    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502625    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502630    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502635    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502641    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502645    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502650    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502654    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502658    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502662    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502666    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:30.515286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502670    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502674    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502679    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502683    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502687    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502691    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502696    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502702    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:30.502706    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.502789    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.505886    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.507020    2574 server.go:1014] "Starting client certificate rotation"
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.507146    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.507174    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.509767    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:19:30.515750 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.514232    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:19:30.525415 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.525401    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:19:30.528528 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.528513    2574 log.go:25] "Validated CRI v1 image API"
Oct 31 01:19:30.529465 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.529437    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:19:30.536648 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.536620    2574 fs.go:135] Filesystem UUIDs: map[1836a17a-b10e-46e9-b6f5-0e264596ab75:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cd2541a1-10e3-4f23-964c-0265e0f797d9:/dev/nvme0n1p3]
Oct 31 01:19:30.536760 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.536648    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 01:19:30.541952 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.541831    2574 manager.go:217] Machine: {Timestamp:2026-10-31 01:19:30.540514081 +0000 UTC m=+0.378637962 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3139925 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec241ee80ca73c3fbef4aa5d15275a6b SystemUUID:ec241ee8-0ca7-3c3f-bef4-aa5d15275a6b BootID:eb47b728-9a73-45df-aaf5-b0c768ed0019 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:6c:3a:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:6c:3a:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:14:c4:72:84:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:19:30.541952 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.541950    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:19:30.542157 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.542072    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:19:30.545832 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.545793    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:19:30.546028 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.545834    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-49.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:19:30.546098 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.546032    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:19:30.546098 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.546061    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:19:30.546098 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.546074    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:19:30.547069 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.547031    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:19:30.548353 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.548341    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:19:30.552930 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.552919    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:19:30.553063 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.553053    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:19:30.553909 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.553899    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:19:30.553939 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.553914    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:19:30.553939 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.553926    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:19:30.553939 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.553935    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:19:30.554029 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.553943    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:19:30.554277 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.554263    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:19:30.556716 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.556696    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:19:30.563093 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.563071    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:19:30.564384 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.564367    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:19:30.565339 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.565319    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:19:30.565407 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.565350    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:19:30.565502 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.565492    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:19:30.565549 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.565508    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:19:30.565549 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.565514    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:19:30.565549 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.565520    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:19:30.565549 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.565526    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:19:30.565549 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.565531    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:19:30.566312 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.566302    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:19:30.566312 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.566311    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:19:30.566381 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.566319    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:19:30.566381 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.566324    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:19:30.566381 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.566333    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:19:30.566381 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.566342    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:19:30.567215 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.567203    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:19:30.567215 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.567214    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:19:30.571295 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.571282    2574 server.go:1263] "Started kubelet"
Oct 31 01:19:30.571429 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.571353    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:19:30.571504 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.571377    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:19:30.571547 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.571530    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:19:30.572030 ip-10-0-130-49 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:19:30.573436 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.573417    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-49.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 01:19:30.573541 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.573490    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 01:19:30.573610 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.573505    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-49.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 01:19:30.573676 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.573609    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:19:30.573976 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.573956    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:19:30.577650 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.577624    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 01:19:30.579912 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.579897    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:19:30.580560 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.580539    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:19:30.580690 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.580673    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:30.581333 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.581318    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:19:30.581426 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.581337    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:19:30.581426 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.581318    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:19:30.581525 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.581474    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:19:30.581525 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.581482    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:19:30.581950 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.580788    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-49.us-east-2.compute.internal.18e37880a6af6b24  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-49.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-49.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:30.571250468 +0000 UTC m=+0.409374349,LastTimestamp:2026-10-31 01:19:30.571250468 +0000 UTC m=+0.409374349,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-49.us-east-2.compute.internal,}"
Oct 31 01:19:30.582010 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.581995    2574 factory.go:55] Registering systemd factory
Oct 31 01:19:30.582010 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.582008    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:19:30.582264 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.582249    2574 factory.go:153] Registering CRI-O factory
Oct 31 01:19:30.582302 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.582268    2574 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:19:30.582328 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.582311    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:19:30.582357 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.582327    2574 factory.go:103] Registering Raw factory
Oct 31 01:19:30.582357 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.582339    2574 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:19:30.583554 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.583484    2574 manager.go:319] Starting recovery of all containers
Oct 31 01:19:30.587977 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.587819    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-49.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 01:19:30.590453 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.590413    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 01:19:30.590973 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.590934    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2d8ll"
Oct 31 01:19:30.594683 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.594670    2574 manager.go:324] Recovery completed
Oct 31 01:19:30.598494 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.598482    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:30.600324 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.600308    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2d8ll"
Oct 31 01:19:30.600406 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.600336    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:30.600406 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.600374    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:30.600406 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.600390    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:30.601263 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.601245    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:19:30.601263 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.601260    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:19:30.601389 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.601278    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:19:30.602471 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.601680    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:30.602728 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.602711    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:30.602830 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.602740    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:30.602830 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.602756    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:30.603652 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.603587    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-49.us-east-2.compute.internal.18e37880a86b7ceb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-49.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-49.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-49.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:30.600353003 +0000 UTC m=+0.438476890,LastTimestamp:2026-10-31 01:19:30.600353003 +0000 UTC m=+0.438476890,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-49.us-east-2.compute.internal,}"
Oct 31 01:19:30.604125 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.604114    2574 policy_none.go:50] "Start"
Oct 31 01:19:30.604180 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.604130    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:19:30.604180 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.604140    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:19:30.606500 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.606489    2574 policy_none.go:44] "Start"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.655614    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.658236    2574 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.658272    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.658284    2574 server.go:87] "Starting device plugin registration server"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.658550    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.658563    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.658720    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.658786    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.658794    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.659313    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.659365    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:30.673291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.672764    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:19:30.675682 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.674176    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:19:30.675682 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.674196    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:19:30.675682 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.674214    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:19:30.675682 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.674248    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 01:19:30.676928 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.676912    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:30.759565 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.759391    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:30.760330 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.760315    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:30.760387 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.760345    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:30.760387 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.760355    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:30.760387 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.760379    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.768713 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.768699    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.768772 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.768721    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-49.us-east-2.compute.internal\": node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:30.774497 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.774476    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"]
Oct 31 01:19:30.774584 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.774526    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:30.775301 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.775288    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:30.775354 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.775317    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:30.775354 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.775333    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:30.777583 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.777572    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:30.777725 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.777713    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.777757 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.777740    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:30.778313 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.778297    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:30.778379 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.778322    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:30.778379 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.778338    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:30.778379 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.778347    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:30.778379 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.778363    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:30.778379 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.778375    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:30.780529 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.780507    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.780609 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.780540    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:30.781218 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.781202    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:30.781287 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.781228    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:30.781287 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.781238    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:30.782576 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.782559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.782658 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.782582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.782658 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.782610    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3e5d96baa84771b6810c7d4e7a868fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"b3e5d96baa84771b6810c7d4e7a868fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.785596 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.785581    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:30.809779 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.809764    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found" node="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.812364 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.812347    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found" node="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.882898 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.882873    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.882987 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.882899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3e5d96baa84771b6810c7d4e7a868fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"b3e5d96baa84771b6810c7d4e7a868fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.882987 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.882915    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.882987 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.882973    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.883114 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.882988    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3e5d96baa84771b6810c7d4e7a868fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"b3e5d96baa84771b6810c7d4e7a868fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.883114 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:30.882973    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:30.885996 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.885979    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:30.986863 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:30.986822    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:31.087741 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:31.087669    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:31.115837 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.115813    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:31.117969 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.117955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:31.188161 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:31.188135    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:31.288780 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:31.288740    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:31.389261 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:31.389195    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:31.489820 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:31.489790    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:31.506938 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.506920    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 01:19:31.507079 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.507063    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 01:19:31.590179 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:31.590157    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:19:31.598647 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.598628    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:31.606243 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.606197    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 01:14:31 +0000 UTC" deadline="2028-07-31 12:48:49.061743314 +0000 UTC"
Oct 31 01:19:31.606243 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.606236    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15347h29m17.455510346s"
Oct 31 01:19:31.681089 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.681066    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:31.695084 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.695060    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:19:31.697051 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.697020    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:19:31.707935 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.707915    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:19:31.778473 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:31.778443    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e5d96baa84771b6810c7d4e7a868fe.slice/crio-dac0236c1f9a8755121e5ddf0aad3212802c1f6e8ac830ba13e4e22d27288aa9 WatchSource:0}: Error finding container dac0236c1f9a8755121e5ddf0aad3212802c1f6e8ac830ba13e4e22d27288aa9: Status 404 returned error can't find the container with id dac0236c1f9a8755121e5ddf0aad3212802c1f6e8ac830ba13e4e22d27288aa9
Oct 31 01:19:31.778865 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:31.778850    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36f9ad789e8b4baae6a17685ca5cf7f.slice/crio-1d4693ff8bd909d8ccb79431d239f4dce367d4180e7db58b9a30cb8a127fa23c WatchSource:0}: Error finding container 1d4693ff8bd909d8ccb79431d239f4dce367d4180e7db58b9a30cb8a127fa23c: Status 404 returned error can't find the container with id 1d4693ff8bd909d8ccb79431d239f4dce367d4180e7db58b9a30cb8a127fa23c
Oct 31 01:19:31.783377 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:31.783358    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:19:32.023484 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.023416    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:32.088739 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.088713    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:32.555099 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.555026    2574 apiserver.go:52] "Watching apiserver"
Oct 31 01:19:32.569340 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.569311    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:19:32.569735 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.569713    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d","openshift-cluster-node-tuning-operator/tuned-kl48l","openshift-multus/network-metrics-daemon-4drdm","openshift-network-diagnostics/network-check-target-67sdw","openshift-network-operator/iptables-alerter-fp4dj","kube-system/global-pull-secret-syncer-sgnh6","openshift-image-registry/node-ca-mg8qc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-b2jz4","kube-system/konnectivity-agent-c9rzr","kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal","openshift-dns/node-resolver-jp6vv","openshift-multus/multus-additional-cni-plugins-ncbph","openshift-multus/multus-gvxxk"]
Oct 31 01:19:32.572450 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.572425    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.574826 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.574784    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.577219 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.577198    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:32.577320 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.577280    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:32.578694 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.578669    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vj55m\""
Oct 31 01:19:32.580016 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.579998    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h7598\""
Oct 31 01:19:32.580134 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.580068    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:32.580202 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.580133    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:32.580202 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.580192    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 01:19:32.583249 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.582481    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.583857 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.583815    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:19:32.583972 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.583879    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:19:32.583972 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.583896    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:19:32.583972 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.583963    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:19:32.584150 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.584037    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:19:32.584314 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.584295    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:19:32.584429 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.584415    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:19:32.586216 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.586195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.586335 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.586299    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:32.586444 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.586201    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:19:32.588899 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.588883    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.589571 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b074486-2077-4e57-8223-f51a2bf0bea3-host-slash\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.589661 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/471cd70e-c62e-4f48-903d-1d71fa617d35-host\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.589661 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589612    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-cnibin\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.589661 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.589866 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.589866 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bcz\" (UniqueName: \"kubernetes.io/projected/7f8bac4d-65fb-4079-ad54-a762105a2d93-kube-api-access-s2bcz\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.589866 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589762    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rskls\" (UniqueName: \"kubernetes.io/projected/0b074486-2077-4e57-8223-f51a2bf0bea3-kube-api-access-rskls\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.589866 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589805    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-system-cni-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.589866 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.590151 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:32.590151 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:32.590151 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589925    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/471cd70e-c62e-4f48-903d-1d71fa617d35-serviceca\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.590151 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589959    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f499c\" (UniqueName: \"kubernetes.io/projected/71d44c81-99b9-4df8-982a-98edaa068b3d-kube-api-access-f499c\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:32.590151 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.589990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b074486-2077-4e57-8223-f51a2bf0bea3-iptables-alerter-script\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.590151 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.590023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b85m\" (UniqueName: \"kubernetes.io/projected/471cd70e-c62e-4f48-903d-1d71fa617d35-kube-api-access-2b85m\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.590456 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.590049    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.591434 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.591411    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.593567 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.593547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.596774 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.596751    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:32.599245 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.599222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.601704 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.601684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.644378 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.644352    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:19:32.644477 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.644432    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z6rqx"
Oct 31 01:19:32.657239 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.656839    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z6rqx"
Oct 31 01:19:32.683986 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.683961    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:19:32.684139 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.683943    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"b3e5d96baa84771b6810c7d4e7a868fe","Type":"ContainerStarted","Data":"dac0236c1f9a8755121e5ddf0aad3212802c1f6e8ac830ba13e4e22d27288aa9"}
Oct 31 01:19:32.685317 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.685287    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"d36f9ad789e8b4baae6a17685ca5cf7f","Type":"ContainerStarted","Data":"1d4693ff8bd909d8ccb79431d239f4dce367d4180e7db58b9a30cb8a127fa23c"}
Oct 31 01:19:32.689650 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.689632    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:19:32.689869 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.689851    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:19:32.689961 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.689895    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:19:32.690754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.690715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-host\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.690868 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.690755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-tmp\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.690868 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.690780    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-ovn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.690868 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.690812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.690868 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.690849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-slash\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.691013 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.690942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-netns\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.691013 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.690991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-daemon-config\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.691109 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691018    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-multus-certs\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.691109 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-etc-kubernetes\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.691227 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691108    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.691227 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:32.691227 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-systemd-units\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.691382 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-var-lib-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.691382 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.691382 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691284    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swnn\" (UniqueName: \"kubernetes.io/projected/4749d5cb-885e-418c-82ff-55ce8029e532-kube-api-access-7swnn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.691382 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.691382 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rskls\" (UniqueName: \"kubernetes.io/projected/0b074486-2077-4e57-8223-f51a2bf0bea3-kube-api-access-rskls\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.691382 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691362    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-netd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691483    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-kubernetes\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691516    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-lib-modules\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cni-binary-copy\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691566    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-conf-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-conf\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691606    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cnibin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b074486-2077-4e57-8223-f51a2bf0bea3-iptables-alerter-script\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.691685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691668    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b85m\" (UniqueName: \"kubernetes.io/projected/471cd70e-c62e-4f48-903d-1d71fa617d35-kube-api-access-2b85m\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-agent-certs\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-tmp-dir\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-modprobe-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/471cd70e-c62e-4f48-903d-1d71fa617d35-host\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691841    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bcz\" (UniqueName: \"kubernetes.io/projected/7f8bac4d-65fb-4079-ad54-a762105a2d93-kube-api-access-s2bcz\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-config\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691968    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-socket-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692035    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692080    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/471cd70e-c62e-4f48-903d-1d71fa617d35-host\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692139    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-tuned\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.692238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-system-cni-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692253    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-systemd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-node-log\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.692342    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-bin\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.692416    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs podName:71d44c81-99b9-4df8-982a-98edaa068b3d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:33.192381832 +0000 UTC m=+3.030505705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs") pod "network-metrics-daemon-4drdm" (UID: "71d44c81-99b9-4df8-982a-98edaa068b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-script-lib\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692467    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692495    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9bh\" (UniqueName: \"kubernetes.io/projected/3fd62759-3b38-496d-b853-9d7bde9eb09d-kube-api-access-zp9bh\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692519    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-socket-dir-parent\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-konnectivity-ca\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-sys\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-netns\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692646    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692694    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692729    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:19:32.692845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692839    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692886    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692923    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692991    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692645    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jmj\" (UniqueName: \"kubernetes.io/projected/5b391665-5177-4a9c-adfb-d8d9d2772ab9-kube-api-access-58jmj\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-etc-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-dbus\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b074486-2077-4e57-8223-f51a2bf0bea3-host-slash\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693144    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-cnibin\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-env-overrides\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693210    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-cnibin\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-etc-selinux\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-bin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-kubelet\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693286    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b074486-2077-4e57-8223-f51a2bf0bea3-host-slash\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693299    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.691936    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.693326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-registration-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.693478 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.692439    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-system-cni-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.694073 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-hostroot\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.694154 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/471cd70e-c62e-4f48-903d-1d71fa617d35-serviceca\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.694265 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694024    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b074486-2077-4e57-8223-f51a2bf0bea3-iptables-alerter-script\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.694265 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f499c\" (UniqueName: \"kubernetes.io/projected/71d44c81-99b9-4df8-982a-98edaa068b3d-kube-api-access-f499c\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:32.694339 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-systemd\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.694339 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694323    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.694410 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmn4x\" (UniqueName: \"kubernetes.io/projected/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-kube-api-access-dmn4x\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.694410 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-system-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.694523 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-k8s-cni-cncf-io\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.694523 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.694523 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.694634 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-hosts-file\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.694634 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-multus\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.694634 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-kubelet\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.694634 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694624    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-log-socket\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.694787 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694631    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/471cd70e-c62e-4f48-903d-1d71fa617d35-serviceca\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.694787 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4749d5cb-885e-418c-82ff-55ce8029e532-ovn-node-metrics-cert\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.694866 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694806    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-device-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.694866 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-sys-fs\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.694933 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694870    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnw7z\" (UniqueName: \"kubernetes.io/projected/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kube-api-access-tnw7z\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.694933 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-kubelet-config\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.694998 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysconfig\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.694998 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-run\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.694998 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.694987    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-var-lib-kubelet\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.695138 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.695011    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:19:32.695374 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.695341    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:19:32.695479 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.695392    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:19:32.695722 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.695700    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.698178 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.698156    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnh8f\""
Oct 31 01:19:32.698178 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.698173    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d5hw4\""
Oct 31 01:19:32.698374 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.698250    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gv7wl\""
Oct 31 01:19:32.698374 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.698251    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm2lp\""
Oct 31 01:19:32.698374 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.698294    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vjbgx\""
Oct 31 01:19:32.698374 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.698366    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tsbmr\""
Oct 31 01:19:32.698560 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.698370    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:19:32.698560 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.698441    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82mbc\""
Oct 31 01:19:32.706864 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.706844    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:19:32.707376 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.707357    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:32.707376 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.707398    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:32.707376 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.707410    2574 projected.go:196] Error preparing data for projected volume kube-api-access-59gbm for pod openshift-network-diagnostics/network-check-target-67sdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:32.707614 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.707493    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm podName:e0826e21-1b23-4f42-a0af-eb1621a858f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:33.207476712 +0000 UTC m=+3.045600603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-59gbm" (UniqueName: "kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm") pod "network-check-target-67sdw" (UID: "e0826e21-1b23-4f42-a0af-eb1621a858f5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:32.711772 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.711748    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rskls\" (UniqueName: \"kubernetes.io/projected/0b074486-2077-4e57-8223-f51a2bf0bea3-kube-api-access-rskls\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.711859 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.711775    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f499c\" (UniqueName: \"kubernetes.io/projected/71d44c81-99b9-4df8-982a-98edaa068b3d-kube-api-access-f499c\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:32.712207 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.712191    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b85m\" (UniqueName: \"kubernetes.io/projected/471cd70e-c62e-4f48-903d-1d71fa617d35-kube-api-access-2b85m\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.712254 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.712205    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bcz\" (UniqueName: \"kubernetes.io/projected/7f8bac4d-65fb-4079-ad54-a762105a2d93-kube-api-access-s2bcz\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.789058 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.789018    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:32.795393 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-log-socket\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.795535 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4749d5cb-885e-418c-82ff-55ce8029e532-ovn-node-metrics-cert\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.795535 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-device-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.795535 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-sys-fs\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.795535 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795495    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-log-socket\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.795535 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnw7z\" (UniqueName: \"kubernetes.io/projected/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kube-api-access-tnw7z\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-kubelet-config\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysconfig\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-run\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795660    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-kubelet-config\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795603    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-device-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795684    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysconfig\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-var-lib-kubelet\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795718    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-var-lib-kubelet\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795633    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-sys-fs\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-host\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-tmp\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795779    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-host\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.795799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-ovn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795812    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-run\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-slash\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795859    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-ovn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795861    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-netns\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-daemon-config\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795905    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-netns\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795918    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-slash\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-multus-certs\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-etc-kubernetes\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.795971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-systemd-units\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796003    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-systemd-units\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796008    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-multus-certs\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796031    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-etc-kubernetes\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-var-lib-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swnn\" (UniqueName: \"kubernetes.io/projected/4749d5cb-885e-418c-82ff-55ce8029e532-kube-api-access-7swnn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796126    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-var-lib-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.796514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796175    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-netd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796171    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796215    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-kubernetes\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-lib-modules\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796243    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-netd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cni-binary-copy\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796291    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-conf-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796308    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-conf\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796309    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-kubernetes\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cnibin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-agent-certs\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796359    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-conf-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796364    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-lib-modules\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796375    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-tmp-dir\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-modprobe-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796421    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.797110 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-config\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796469    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-socket-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-tuned\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796497    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-conf\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-systemd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-node-log\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796553    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-modprobe-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796568    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-bin\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-script-lib\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9bh\" (UniqueName: \"kubernetes.io/projected/3fd62759-3b38-496d-b853-9d7bde9eb09d-kube-api-access-zp9bh\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-socket-dir-parent\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-konnectivity-ca\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-sys\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-netns\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jmj\" (UniqueName: \"kubernetes.io/projected/5b391665-5177-4a9c-adfb-d8d9d2772ab9-kube-api-access-58jmj\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798002 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-etc-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-dbus\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-env-overrides\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796907    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-etc-selinux\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-bin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.796965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-kubelet\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-registration-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-hostroot\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-systemd\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797084    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmn4x\" (UniqueName: \"kubernetes.io/projected/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-kube-api-access-dmn4x\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-system-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-k8s-cni-cncf-io\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797271    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-hosts-file\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-multus\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797319    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-daemon-config\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.798855 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797334    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-kubelet\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797494    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-socket-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797629    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-dbus\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.797963    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cnibin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798153    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-tmp-dir\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798301    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cni-binary-copy\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798436    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-etc-selinux\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798551    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-socket-dir-parent\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798556    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-bin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798584    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-kubelet\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798621    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-hostroot\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798637    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-config\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798671    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-systemd\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798695    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-k8s-cni-cncf-io\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798708    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798723    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-netns\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798756    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-sys\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.799720 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798882    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-hosts-file\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798924    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-systemd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798951    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-node-log\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.798974    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-bin\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.799097    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-konnectivity-ca\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.799178    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-registration-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.799236    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.799351    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-tmp\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.799394    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.799407    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-script-lib\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.799420    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:32.799454    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret podName:552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:33.299435828 +0000 UTC m=+3.137559699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret") pod "global-pull-secret-syncer-sgnh6" (UID: "552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.799861    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-multus\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.800443 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.799919    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-env-overrides\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.801360 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.801330    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-etc-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.801508 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.801496    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-kubelet\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.801568 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.801528    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-system-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.801904 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.801879    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-agent-certs\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:32.803178 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.803156    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-tuned\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.803496 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.803468    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4749d5cb-885e-418c-82ff-55ce8029e532-ovn-node-metrics-cert\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.816107 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.816034    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnw7z\" (UniqueName: \"kubernetes.io/projected/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kube-api-access-tnw7z\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.818451 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.818429    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jmj\" (UniqueName: \"kubernetes.io/projected/5b391665-5177-4a9c-adfb-d8d9d2772ab9-kube-api-access-58jmj\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.820730 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.820703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9bh\" (UniqueName: \"kubernetes.io/projected/3fd62759-3b38-496d-b853-9d7bde9eb09d-kube-api-access-zp9bh\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.820887 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.820830    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swnn\" (UniqueName: \"kubernetes.io/projected/4749d5cb-885e-418c-82ff-55ce8029e532-kube-api-access-7swnn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.822661 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.822630    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmn4x\" (UniqueName: \"kubernetes.io/projected/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-kube-api-access-dmn4x\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.892824 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.892794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:19:32.903779 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.903747    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471cd70e_c62e_4f48_903d_1d71fa617d35.slice/crio-bd0dd9ba261dee9861e3af7fb84f797f612bf9475cd15d09ca0eba4d78077789 WatchSource:0}: Error finding container bd0dd9ba261dee9861e3af7fb84f797f612bf9475cd15d09ca0eba4d78077789: Status 404 returned error can't find the container with id bd0dd9ba261dee9861e3af7fb84f797f612bf9475cd15d09ca0eba4d78077789
Oct 31 01:19:32.916475 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.916453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:19:32.921178 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.921156    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:19:32.928144 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.928106    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8bac4d_65fb_4079_ad54_a762105a2d93.slice/crio-02e96a9b79e9dba006fbe6548382d8dfb674f7620a2085b17b620875713f2df0 WatchSource:0}: Error finding container 02e96a9b79e9dba006fbe6548382d8dfb674f7620a2085b17b620875713f2df0: Status 404 returned error can't find the container with id 02e96a9b79e9dba006fbe6548382d8dfb674f7620a2085b17b620875713f2df0
Oct 31 01:19:32.932967 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.932530    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b074486_2077_4e57_8223_f51a2bf0bea3.slice/crio-635ac6929582ced57b84c752fb07c0f1892859ad65d4be1dc595d16533337047 WatchSource:0}: Error finding container 635ac6929582ced57b84c752fb07c0f1892859ad65d4be1dc595d16533337047: Status 404 returned error can't find the container with id 635ac6929582ced57b84c752fb07c0f1892859ad65d4be1dc595d16533337047
Oct 31 01:19:32.947696 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.947674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:32.950182 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.950160    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:19:32.952205 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.952104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:19:32.955624 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.955598    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:32.960250 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.960172    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4749d5cb_885e_418c_82ff_55ce8029e532.slice/crio-3920eea7457abf4cbc916b6eb03a38f78a18061dc99e826ab98fe57b259bb478 WatchSource:0}: Error finding container 3920eea7457abf4cbc916b6eb03a38f78a18061dc99e826ab98fe57b259bb478: Status 404 returned error can't find the container with id 3920eea7457abf4cbc916b6eb03a38f78a18061dc99e826ab98fe57b259bb478
Oct 31 01:19:32.962941 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.962919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gvxxk"
Oct 31 01:19:32.966557 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.966517    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd62759_3b38_496d_b853_9d7bde9eb09d.slice/crio-9b145da0cec4818e8fa8e1bbc1661eed975cdf1c654f1fbd6bc60f5fb1b8fc29 WatchSource:0}: Error finding container 9b145da0cec4818e8fa8e1bbc1661eed975cdf1c654f1fbd6bc60f5fb1b8fc29: Status 404 returned error can't find the container with id 9b145da0cec4818e8fa8e1bbc1661eed975cdf1c654f1fbd6bc60f5fb1b8fc29
Oct 31 01:19:32.971152 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:32.970597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:19:32.976059 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.975527    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0eaa17_d3d8_4db5_b63c_1d74a7d77996.slice/crio-0b92ed9766035f2143dc143b1cd43b5a568dba8b24897e79a0907c03a5f07103 WatchSource:0}: Error finding container 0b92ed9766035f2143dc143b1cd43b5a568dba8b24897e79a0907c03a5f07103: Status 404 returned error can't find the container with id 0b92ed9766035f2143dc143b1cd43b5a568dba8b24897e79a0907c03a5f07103
Oct 31 01:19:32.977845 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.977823    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0e8c4d_8ba4_4f1b_914d_0d7b17ec7406.slice/crio-9d90f2c4d59c098e522caa5577fedc67736a0a5eab95555956da3583f8c39655 WatchSource:0}: Error finding container 9d90f2c4d59c098e522caa5577fedc67736a0a5eab95555956da3583f8c39655: Status 404 returned error can't find the container with id 9d90f2c4d59c098e522caa5577fedc67736a0a5eab95555956da3583f8c39655
Oct 31 01:19:32.979432 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.979406    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b391665_5177_4a9c_adfb_d8d9d2772ab9.slice/crio-91a120bc27513dade25956dde8909d486d7728987442391f9ddf58171ba7a6db WatchSource:0}: Error finding container 91a120bc27513dade25956dde8909d486d7728987442391f9ddf58171ba7a6db: Status 404 returned error can't find the container with id 91a120bc27513dade25956dde8909d486d7728987442391f9ddf58171ba7a6db
Oct 31 01:19:32.987304 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:19:32.987280    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4950850b_9fc3_4c93_a3c8_b92ef5d906cc.slice/crio-58405bcd9e73457678a0eac63a61fabfd983ac65db7e3adbb7c1b8c6ff2369e3 WatchSource:0}: Error finding container 58405bcd9e73457678a0eac63a61fabfd983ac65db7e3adbb7c1b8c6ff2369e3: Status 404 returned error can't find the container with id 58405bcd9e73457678a0eac63a61fabfd983ac65db7e3adbb7c1b8c6ff2369e3
Oct 31 01:19:33.200543 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.200390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:33.200708 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:33.200586    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:33.200708 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:33.200665    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs podName:71d44c81-99b9-4df8-982a-98edaa068b3d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:34.200643428 +0000 UTC m=+4.038767315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs") pod "network-metrics-daemon-4drdm" (UID: "71d44c81-99b9-4df8-982a-98edaa068b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:33.301560 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.301520    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:33.301727 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.301579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:33.301727 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:33.301702    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:33.301727 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:33.301716    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:33.301914 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:33.301734    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:33.301914 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:33.301747    2574 projected.go:196] Error preparing data for projected volume kube-api-access-59gbm for pod openshift-network-diagnostics/network-check-target-67sdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:33.301914 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:33.301788    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret podName:552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:34.301767072 +0000 UTC m=+4.139890947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret") pod "global-pull-secret-syncer-sgnh6" (UID: "552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:33.301914 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:33.301808    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm podName:e0826e21-1b23-4f42-a0af-eb1621a858f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:34.301800047 +0000 UTC m=+4.139923917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-59gbm" (UniqueName: "kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm") pod "network-check-target-67sdw" (UID: "e0826e21-1b23-4f42-a0af-eb1621a858f5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:33.660311 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.660209    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:33 +0000 UTC" deadline="2028-04-24 21:06:22.35420411 +0000 UTC"
Oct 31 01:19:33.660311 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.660273    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13003h46m48.693936092s"
Oct 31 01:19:33.693773 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.693659    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gvxxk" event={"ID":"5b391665-5177-4a9c-adfb-d8d9d2772ab9","Type":"ContainerStarted","Data":"91a120bc27513dade25956dde8909d486d7728987442391f9ddf58171ba7a6db"}
Oct 31 01:19:33.698659 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.698627    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c9rzr" event={"ID":"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996","Type":"ContainerStarted","Data":"0b92ed9766035f2143dc143b1cd43b5a568dba8b24897e79a0907c03a5f07103"}
Oct 31 01:19:33.705705 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.705668    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d" event={"ID":"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406","Type":"ContainerStarted","Data":"9d90f2c4d59c098e522caa5577fedc67736a0a5eab95555956da3583f8c39655"}
Oct 31 01:19:33.708378 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.708346    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kl48l" event={"ID":"3fd62759-3b38-496d-b853-9d7bde9eb09d","Type":"ContainerStarted","Data":"9b145da0cec4818e8fa8e1bbc1661eed975cdf1c654f1fbd6bc60f5fb1b8fc29"}
Oct 31 01:19:33.719042 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.718942    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"3920eea7457abf4cbc916b6eb03a38f78a18061dc99e826ab98fe57b259bb478"}
Oct 31 01:19:33.721799 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.721738    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerStarted","Data":"02e96a9b79e9dba006fbe6548382d8dfb674f7620a2085b17b620875713f2df0"}
Oct 31 01:19:33.729287 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.729256    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fp4dj" event={"ID":"0b074486-2077-4e57-8223-f51a2bf0bea3","Type":"ContainerStarted","Data":"635ac6929582ced57b84c752fb07c0f1892859ad65d4be1dc595d16533337047"}
Oct 31 01:19:33.738428 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.738404    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mg8qc" event={"ID":"471cd70e-c62e-4f48-903d-1d71fa617d35","Type":"ContainerStarted","Data":"bd0dd9ba261dee9861e3af7fb84f797f612bf9475cd15d09ca0eba4d78077789"}
Oct 31 01:19:33.743090 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:33.743065    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jp6vv" event={"ID":"4950850b-9fc3-4c93-a3c8-b92ef5d906cc","Type":"ContainerStarted","Data":"58405bcd9e73457678a0eac63a61fabfd983ac65db7e3adbb7c1b8c6ff2369e3"}
Oct 31 01:19:34.209496 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.209456    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:34.209683 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.209665    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:34.209752 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.209740    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs podName:71d44c81-99b9-4df8-982a-98edaa068b3d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:36.209719839 +0000 UTC m=+6.047843725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs") pod "network-metrics-daemon-4drdm" (UID: "71d44c81-99b9-4df8-982a-98edaa068b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:34.312340 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.312307    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:34.312466 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.312365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:34.312526 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.312504    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:34.312581 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.312526    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:34.312581 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.312539    2574 projected.go:196] Error preparing data for projected volume kube-api-access-59gbm for pod openshift-network-diagnostics/network-check-target-67sdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:34.312675 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.312596    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm podName:e0826e21-1b23-4f42-a0af-eb1621a858f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:36.312576789 +0000 UTC m=+6.150700661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-59gbm" (UniqueName: "kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm") pod "network-check-target-67sdw" (UID: "e0826e21-1b23-4f42-a0af-eb1621a858f5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:34.313044 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.313020    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:34.313171 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.313081    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret podName:552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:36.313065203 +0000 UTC m=+6.151189074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret") pod "global-pull-secret-syncer-sgnh6" (UID: "552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:34.661404 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.661254    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:33 +0000 UTC" deadline="2028-05-29 01:41:45.291435678 +0000 UTC"
Oct 31 01:19:34.661404 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.661320    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13824h22m10.630120002s"
Oct 31 01:19:34.679301 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.678783    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:34.679301 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.678916    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:34.679301 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.679000    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:34.679301 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.679067    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:34.681275 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.681216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:34.681409 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:34.681352    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:34.760368 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.760321    2574 generic.go:356] "Generic (PLEG): container finished" podID="d36f9ad789e8b4baae6a17685ca5cf7f" containerID="82dd953d8c031bf46fef706237fae69f65cff0fa5be2272a97d9957016d12278" exitCode=0
Oct 31 01:19:34.760701 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.760671    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"d36f9ad789e8b4baae6a17685ca5cf7f","Type":"ContainerDied","Data":"82dd953d8c031bf46fef706237fae69f65cff0fa5be2272a97d9957016d12278"}
Oct 31 01:19:34.781773 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.781292    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"b3e5d96baa84771b6810c7d4e7a868fe","Type":"ContainerStarted","Data":"99273834aa319291a8a9f6a37a236859e7cc8012895d41a186c837087a7c41f2"}
Oct 31 01:19:34.803652 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:34.803514    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal" podStartSLOduration=2.803496973 podStartE2EDuration="2.803496973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:34.802755631 +0000 UTC m=+4.640879554" watchObservedRunningTime="2026-10-31 01:19:34.803496973 +0000 UTC m=+4.641620871"
Oct 31 01:19:35.794403 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:35.794369    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"d36f9ad789e8b4baae6a17685ca5cf7f","Type":"ContainerStarted","Data":"5eb99b666d36a8cd580c29d16d540b7d3f9c335b416b9ece865a4de83d19d267"}
Oct 31 01:19:36.230160 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.230127    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:36.230312 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.230291    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:36.230385 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.230368    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs podName:71d44c81-99b9-4df8-982a-98edaa068b3d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:40.230348753 +0000 UTC m=+10.068472657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs") pod "network-metrics-daemon-4drdm" (UID: "71d44c81-99b9-4df8-982a-98edaa068b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:36.331040 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.330474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:36.331040 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.330537    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:36.331040 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.330716    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:36.331040 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.330738    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:36.331040 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.330751    2574 projected.go:196] Error preparing data for projected volume kube-api-access-59gbm for pod openshift-network-diagnostics/network-check-target-67sdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:36.331040 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.330812    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm podName:e0826e21-1b23-4f42-a0af-eb1621a858f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:40.330793658 +0000 UTC m=+10.168917542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-59gbm" (UniqueName: "kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm") pod "network-check-target-67sdw" (UID: "e0826e21-1b23-4f42-a0af-eb1621a858f5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:36.331040 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.330886    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:36.341137 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.331889    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret podName:552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:40.330909039 +0000 UTC m=+10.169032924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret") pod "global-pull-secret-syncer-sgnh6" (UID: "552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:36.675267 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.675185    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:36.675425 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.675323    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:36.675754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.675731    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:36.675867 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.675842    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:36.676064 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.676028    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:36.676206 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:36.676168    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:36.801340 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.801310    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerDied","Data":"5bd8dffaa6837b759185e2972417b4ae3d581f69e2da2f6fd81472658a7d88e9"}
Oct 31 01:19:36.801827 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.801276    2574 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="5bd8dffaa6837b759185e2972417b4ae3d581f69e2da2f6fd81472658a7d88e9" exitCode=0
Oct 31 01:19:36.836243 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:36.836186    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal" podStartSLOduration=4.836168713 podStartE2EDuration="4.836168713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:35.815991328 +0000 UTC m=+5.654115235" watchObservedRunningTime="2026-10-31 01:19:36.836168713 +0000 UTC m=+6.674292608"
Oct 31 01:19:37.812680 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:37.812207    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c9rzr" event={"ID":"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996","Type":"ContainerStarted","Data":"e28a8dad444e05a657c804c7a9d0fe167177e85f751fdcaceecc145f13f7e8ea"}
Oct 31 01:19:37.834299 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:37.834019    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c9rzr" podStartSLOduration=2.749078516 podStartE2EDuration="6.833997961s" totalImagesPullingTime="4.084919445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:37.833495277 +0000 UTC m=+7.671619172" watchObservedRunningTime="2026-10-31 01:19:37.833997961 +0000 UTC m=+7.672121848"
Oct 31 01:19:38.675361 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:38.675323    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:38.675548 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:38.675489    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:38.675906 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:38.675883    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:38.676018 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:38.675985    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:38.676148 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:38.676036    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:38.676148 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:38.676109    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:38.819355 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:38.819273    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mg8qc" event={"ID":"471cd70e-c62e-4f48-903d-1d71fa617d35","Type":"ContainerStarted","Data":"29f096b28af5509e0630c26a95feacc7801343b61f2a20b932f5addda3058f0a"}
Oct 31 01:19:39.823240 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:39.823142    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d" event={"ID":"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406","Type":"ContainerStarted","Data":"b70d1bdc297992c228d2ab90c6340e7434791acfef65b5f309d0ae25dffdffe2"}
Oct 31 01:19:40.264418 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:40.264316    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:40.264576 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.264539    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:40.264651 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.264608    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs podName:71d44c81-99b9-4df8-982a-98edaa068b3d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.264588611 +0000 UTC m=+18.102712497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs") pod "network-metrics-daemon-4drdm" (UID: "71d44c81-99b9-4df8-982a-98edaa068b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:40.364764 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:40.364718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:40.364941 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:40.364788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:40.365021 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.365001    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:40.365074 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.365033    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:40.365074 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.365047    2574 projected.go:196] Error preparing data for projected volume kube-api-access-59gbm for pod openshift-network-diagnostics/network-check-target-67sdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:40.365225 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.365125    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm podName:e0826e21-1b23-4f42-a0af-eb1621a858f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.365105187 +0000 UTC m=+18.203229072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-59gbm" (UniqueName: "kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm") pod "network-check-target-67sdw" (UID: "e0826e21-1b23-4f42-a0af-eb1621a858f5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:40.365572 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.365550    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:40.365653 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.365611    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret podName:552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.365595891 +0000 UTC m=+18.203719761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret") pod "global-pull-secret-syncer-sgnh6" (UID: "552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:40.679679 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:40.678977    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:40.679679 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.679137    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:40.679679 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:40.679527    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:40.679679 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.679604    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:40.680112 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:40.680055    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:40.680394 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:40.680340    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:41.148440 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.148416    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:19:41.352431 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.352397    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:41.353974 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.353689    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:41.383574 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.378725    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mg8qc" podStartSLOduration=5.109823709 podStartE2EDuration="10.378703868s" totalImagesPullingTime="5.268880159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:38.841671045 +0000 UTC m=+8.679794940" watchObservedRunningTime="2026-10-31 01:19:41.378703868 +0000 UTC m=+11.216827773"
Oct 31 01:19:41.669588 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.669496    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:19:41.14843785Z","UUID":"d3a0f92d-9bc9-4911-a31c-780b38810028","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:19:41.671257 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.671234    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:19:41.671326 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.671272    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:19:41.829772 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.829734    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jp6vv" event={"ID":"4950850b-9fc3-4c93-a3c8-b92ef5d906cc","Type":"ContainerStarted","Data":"25d96ee0d787ca95f452316f0e9a651275d9e60372932612a9fcd881a697e6d8"}
Oct 31 01:19:41.832279 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.832239    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gvxxk" event={"ID":"5b391665-5177-4a9c-adfb-d8d9d2772ab9","Type":"ContainerStarted","Data":"f7925d2fa28fa482982ff740318228f8e137860639b3da3f9785133012a567d7"}
Oct 31 01:19:41.835777 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.835753    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d" event={"ID":"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406","Type":"ContainerStarted","Data":"b92a675337527d9c673c9946f29329b378b20f34d94000ec09a7c119ea93f04e"}
Oct 31 01:19:41.838244 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.838216    2574 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="e33292854a98931337733508f89c67c501d5ab2a75a3abc69d2a6b08dd10e7cc" exitCode=0
Oct 31 01:19:41.838380 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.838349    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerDied","Data":"e33292854a98931337733508f89c67c501d5ab2a75a3abc69d2a6b08dd10e7cc"}
Oct 31 01:19:41.884384 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.884305    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jp6vv" podStartSLOduration=2.635220854 podStartE2EDuration="10.884286358s" totalImagesPullingTime="8.249065504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:41.850674995 +0000 UTC m=+11.688798876" watchObservedRunningTime="2026-10-31 01:19:41.884286358 +0000 UTC m=+11.722410284"
Oct 31 01:19:41.908169 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:41.908093    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gvxxk" podStartSLOduration=2.7740345360000003 podStartE2EDuration="10.908077656s" totalImagesPullingTime="8.13404312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:41.90737701 +0000 UTC m=+11.745500905" watchObservedRunningTime="2026-10-31 01:19:41.908077656 +0000 UTC m=+11.746201549"
Oct 31 01:19:42.678951 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:42.678926    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:42.679420 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:42.678972    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:42.679420 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:42.679109    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:42.679552 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:42.679437    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:42.679622 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:42.679542    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:42.679662 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:42.679621    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:42.843079 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:42.843018    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d" event={"ID":"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406","Type":"ContainerStarted","Data":"2de93572bd0b30a50dfd0d6c19b3dfb07dd0239594b965a549a262b5c696ff78"}
Oct 31 01:19:42.845555 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:42.845516    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerStarted","Data":"fdb21c5c81d6c49ba621b29af3cf2bc9dcde24267132a607c6425141bfdf277e"}
Oct 31 01:19:42.869941 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:42.868642    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d" podStartSLOduration=2.933277234 podStartE2EDuration="11.86862461s" totalImagesPullingTime="8.935347376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:42.868224469 +0000 UTC m=+12.706348358" watchObservedRunningTime="2026-10-31 01:19:42.86862461 +0000 UTC m=+12.706748505"
Oct 31 01:19:43.851325 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:43.851085    2574 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="fdb21c5c81d6c49ba621b29af3cf2bc9dcde24267132a607c6425141bfdf277e" exitCode=0
Oct 31 01:19:43.851785 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:43.851174    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerDied","Data":"fdb21c5c81d6c49ba621b29af3cf2bc9dcde24267132a607c6425141bfdf277e"}
Oct 31 01:19:43.855588 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:43.855350    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fp4dj" event={"ID":"0b074486-2077-4e57-8223-f51a2bf0bea3","Type":"ContainerStarted","Data":"a5e7a513cf3bd23044f23309d3a48671d09fd31d3ce190df3655df05645aa80b"}
Oct 31 01:19:43.909579 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:43.909455    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fp4dj" podStartSLOduration=4.336664288 podStartE2EDuration="12.909436629s" totalImagesPullingTime="8.572772341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:43.908779181 +0000 UTC m=+13.746903076" watchObservedRunningTime="2026-10-31 01:19:43.909436629 +0000 UTC m=+13.747560524"
Oct 31 01:19:44.674947 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:44.674917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:44.675179 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:44.675065    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:44.675688 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:44.675668    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:44.675809 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:44.675741    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:44.675887 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:44.675875    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:44.675944 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:44.675931    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:44.860820 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:44.860786    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kl48l" event={"ID":"3fd62759-3b38-496d-b853-9d7bde9eb09d","Type":"ContainerStarted","Data":"e2917c247156e31c4c00cf0bd6a9c871297d87ffa7a7e4cb9708604d38746bca"}
Oct 31 01:19:44.886255 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:44.886207    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kl48l" podStartSLOduration=2.998844161 podStartE2EDuration="13.886188739s" totalImagesPullingTime="10.887344578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:44.885325891 +0000 UTC m=+14.723449785" watchObservedRunningTime="2026-10-31 01:19:44.886188739 +0000 UTC m=+14.724312645"
Oct 31 01:19:45.864317 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:45.864281    2574 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="f814a66700a0ddfff84cc8627bc5ae57d493fe4b231288b66f19d0bb20babe65" exitCode=0
Oct 31 01:19:45.864978 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:45.864372    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerDied","Data":"f814a66700a0ddfff84cc8627bc5ae57d493fe4b231288b66f19d0bb20babe65"}
Oct 31 01:19:46.675106 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:46.675074    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:46.675291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:46.675113    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:46.675291 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:46.675173    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:46.675291 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:46.675275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:46.675448 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:46.675360    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:46.675500 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:46.675445    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:48.331865 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:48.331617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:48.332264 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.331774    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:48.332264 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.331993    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs podName:71d44c81-99b9-4df8-982a-98edaa068b3d nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.331965203 +0000 UTC m=+34.170089083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs") pod "network-metrics-daemon-4drdm" (UID: "71d44c81-99b9-4df8-982a-98edaa068b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:48.432543 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:48.432504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:48.432726 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:48.432562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:48.432726 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.432685    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:48.432814 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.432759    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret podName:552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.432745365 +0000 UTC m=+34.270869247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret") pod "global-pull-secret-syncer-sgnh6" (UID: "552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:48.432814 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.432690    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:48.432814 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.432812    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:48.432958 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.432825    2574 projected.go:196] Error preparing data for projected volume kube-api-access-59gbm for pod openshift-network-diagnostics/network-check-target-67sdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:48.432958 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.432882    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm podName:e0826e21-1b23-4f42-a0af-eb1621a858f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.432867116 +0000 UTC m=+34.270990998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-59gbm" (UniqueName: "kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm") pod "network-check-target-67sdw" (UID: "e0826e21-1b23-4f42-a0af-eb1621a858f5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:48.677845 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:48.677767    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:48.677988 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:48.677886    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:48.677988 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.677905    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:48.677988 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:48.677919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:48.678178 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.678029    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:48.678178 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:48.678129    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:48.874901 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:48.874867    2574 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="88a5364e0cb107bce7a3afde13e245007e2cb49b01092549c88f2cd5e1f3b3ea" exitCode=0
Oct 31 01:19:48.875076 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:48.874906    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerDied","Data":"88a5364e0cb107bce7a3afde13e245007e2cb49b01092549c88f2cd5e1f3b3ea"}
Oct 31 01:19:49.879076 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:49.879026    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"d5142432e7d2d9161487da039f14f96d12790ac86c678a208974edb1ac2ebc2a"}
Oct 31 01:19:49.879462 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:49.879142    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"71f0c2b8eb791edade690703c131ff272a1c4bddcce60e8cce797fe1be1dfe17"}
Oct 31 01:19:49.879462 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:49.879181    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"f788aa27489742bbc1a73b8fe2aefb8ef855a6e697c18f71d0c901ba003098f6"}
Oct 31 01:19:49.882231 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:49.882199    2574 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="c90f3a0044ba0976f7c176e5740bf924b55fef80cd64bf5ff1a61c1039195077" exitCode=0
Oct 31 01:19:49.882303 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:49.882249    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerDied","Data":"c90f3a0044ba0976f7c176e5740bf924b55fef80cd64bf5ff1a61c1039195077"}
Oct 31 01:19:50.384880 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.384651    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:50.385034 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.384906    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:19:50.385273 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.385258    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:19:50.676066 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.675978    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:50.676202 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.676106    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:50.676202 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:50.676150    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:50.676267 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:50.676199    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:50.676267 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.676208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:50.676329 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:50.676269    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:50.886668 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.886635    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"f9ed736f0b7f0b8e3c2c020adf13d47ac4e340c35c03f6c75287433a036e48b0"}
Oct 31 01:19:50.887079 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.886721    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"44e6780553448c2799383212a6c04fb337ca5af051d4c47ee10c89e56af158d8"}
Oct 31 01:19:50.887079 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.886771    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"3b2f8937171ba8ca4e0f50854439b7e4d36a258639316e321b866e49e148976b"}
Oct 31 01:19:50.889982 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.889961    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncbph" event={"ID":"7f8bac4d-65fb-4079-ad54-a762105a2d93","Type":"ContainerStarted","Data":"f83b14489690237b9aedffa29fc65a792e70f88b2953c13216f04ddc092888a3"}
Oct 31 01:19:50.921700 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:50.921661    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ncbph" podStartSLOduration=7.803236053 podStartE2EDuration="19.921649544s" totalImagesPullingTime="12.118413491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:50.921527734 +0000 UTC m=+20.759651624" watchObservedRunningTime="2026-10-31 01:19:50.921649544 +0000 UTC m=+20.759773433"
Oct 31 01:19:52.675310 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:52.675279    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:52.675310 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:52.675315    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:52.675929 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:52.675368    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:52.675929 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:52.675420    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:52.675929 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:52.675501    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:52.675929 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:52.675550    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:52.896313 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:52.896280    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"906d59b8c0c6d499dbdca7bdcb09a6ddaf24590567f209b85db6b196024021e0"}
Oct 31 01:19:54.675166 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:54.675137    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:54.675861 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:54.675159    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:54.675861 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:54.675253    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:54.675861 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:54.675266    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:54.675861 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:54.675323    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:54.675861 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:54.675384    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:54.904144 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:54.904111    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" event={"ID":"4749d5cb-885e-418c-82ff-55ce8029e532","Type":"ContainerStarted","Data":"e519a98a93e6cd97b1f6eb6e087a69ce9e78c01ee16ac346d22d7fbca6e3102f"}
Oct 31 01:19:54.904405 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:54.904381    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:54.919659 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:54.919640    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:54.944586 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:54.944521    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4" podStartSLOduration=7.210024896 podStartE2EDuration="23.944509913s" totalImagesPullingTime="16.734485017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:54.944065197 +0000 UTC m=+24.782189088" watchObservedRunningTime="2026-10-31 01:19:54.944509913 +0000 UTC m=+24.782633802"
Oct 31 01:19:55.907210 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.907012    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:19:55.907623 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.907161    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:55.923742 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.923720    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:55.957329 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.957302    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sgnh6"]
Oct 31 01:19:55.957463 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.957417    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:55.957555 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:55.957528    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:55.960800 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.960770    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-67sdw"]
Oct 31 01:19:55.960924 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.960860    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:55.960980 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:55.960934    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:55.961410 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.961391    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4drdm"]
Oct 31 01:19:55.961514 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:55.961476    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:55.961569 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:55.961550    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:56.908771 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:56.908742    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:19:57.050676 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:57.050642    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:19:57.675296 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:57.675219    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:57.675445 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:57.675222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:57.675445 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:57.675309    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:57.675445 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:57.675332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:57.675445 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:57.675417    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:57.675603 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:57.675479    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:59.582410 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:59.582371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c9rzr_fb0eaa17-d3d8-4db5-b63c-1d74a7d77996/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:59.675382 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:59.675355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:19:59.675382 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:59.675370    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:19:59.675590 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:59.675491    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:19:59.675590 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:59.675480    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:19:59.675590 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:59.675561    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:19:59.675735 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:19:59.675592    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:19:59.903758 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:19:59.903685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c9rzr_fb0eaa17-d3d8-4db5-b63c-1d74a7d77996/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:00.065588 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:00.065550    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c9rzr_fb0eaa17-d3d8-4db5-b63c-1d74a7d77996/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:00.332312 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:00.332275    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c9rzr_fb0eaa17-d3d8-4db5-b63c-1d74a7d77996/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:00.731618 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:00.731549    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jp6vv_4950850b-9fc3-4c93-a3c8-b92ef5d906cc/dns-node-resolver/0.log"
Oct 31 01:20:01.675053 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:01.675022    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:01.675213 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:01.675114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:01.675213 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:01.675139    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:01.675279 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:01.675213    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:20:01.675316 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:01.675274    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:20:01.675348 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:01.675338    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:20:01.732186 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:01.732157    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mg8qc_471cd70e-c62e-4f48-903d-1d71fa617d35/node-ca/0.log"
Oct 31 01:20:03.674930 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:03.674897    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:03.674930 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:03.674932    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:03.675471 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:03.674991    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:03.675471 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:03.675005    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:20:03.675471 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:03.675076    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:20:03.675471 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:03.675158    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:20:04.341410 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:04.341380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:04.341574 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:04.341504    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:20:04.341574 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:04.341554    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs podName:71d44c81-99b9-4df8-982a-98edaa068b3d nodeName:}" failed. No retries permitted until 2026-10-31 01:20:36.341541915 +0000 UTC m=+66.179665796 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs") pod "network-metrics-daemon-4drdm" (UID: "71d44c81-99b9-4df8-982a-98edaa068b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:20:04.441841 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:04.441814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:04.442008 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:04.441871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:04.442008 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:04.441969    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:20:04.442008 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:04.441983    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:20:04.442008 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:04.442005    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:20:04.442173 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:04.442014    2574 projected.go:196] Error preparing data for projected volume kube-api-access-59gbm for pod openshift-network-diagnostics/network-check-target-67sdw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:20:04.442173 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:04.442024    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret podName:552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:36.442011684 +0000 UTC m=+66.280135564 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret") pod "global-pull-secret-syncer-sgnh6" (UID: "552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:20:04.442173 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:04.442074    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm podName:e0826e21-1b23-4f42-a0af-eb1621a858f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:36.442058334 +0000 UTC m=+66.280182203 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-59gbm" (UniqueName: "kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm") pod "network-check-target-67sdw" (UID: "e0826e21-1b23-4f42-a0af-eb1621a858f5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:20:05.675397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:05.675373    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:05.675784 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:05.675373    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:05.675784 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:05.675483    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:20:05.675784 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:05.675374    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:05.675784 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:05.675542    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:20:05.675784 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:05.675633    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:20:07.675217 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:07.675182    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:07.675217 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:07.675211    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:07.675624 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:07.675287    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:20:07.675624 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:07.675331    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:07.675624 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:07.675401    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:20:07.675624 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:07.675471    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:20:09.675403 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:09.675371    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:09.675403 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:09.675391    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:09.675819 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:09.675516    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:20:09.675819 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:09.675547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:09.675819 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:09.675615    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:20:09.675819 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:09.675664    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:20:11.674487 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:11.674454    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:11.674487 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:11.674469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:11.674983 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:11.674550    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:20:11.674983 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:11.674592    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:11.674983 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:11.674712    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:20:11.674983 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:11.674837    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:20:13.674817 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:13.674785    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:13.675197 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:13.674839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:13.675197 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:13.674851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:13.675197 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:13.674943    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:20:13.675197 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:13.674976    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:20:13.675197 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:13.675079    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:20:15.674570 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:15.674542    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:15.675072 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:15.674573    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:15.675072 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:15.674643    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4drdm" podUID="71d44c81-99b9-4df8-982a-98edaa068b3d"
Oct 31 01:20:15.675072 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:15.674671    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:15.675072 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:15.674724    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sdw" podUID="e0826e21-1b23-4f42-a0af-eb1621a858f5"
Oct 31 01:20:15.675072 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:15.674840    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sgnh6" podUID="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6"
Oct 31 01:20:16.954098 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:16.954074    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeReady"
Oct 31 01:20:16.954480 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:16.954177    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 01:20:17.017381 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.017353    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lxzrw"]
Oct 31 01:20:17.020180 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.020165    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vs2gb"]
Oct 31 01:20:17.020302 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.020286    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:17.023049 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.023024    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.025672 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.025649    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:20:17.025821 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.025807    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:20:17.025986 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.025970    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:20:17.026071 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.026019    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-79nk6\""
Oct 31 01:20:17.036379 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.036361    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vs2gb"]
Oct 31 01:20:17.036989 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.036970    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jtnln"]
Oct 31 01:20:17.039981 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.039963    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lxzrw"]
Oct 31 01:20:17.040088 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.040076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.055172 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.055149    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jtnln"]
Oct 31 01:20:17.126789 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.126753    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:20:17.126975 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.126808    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:20:17.126975 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.126873    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:20:17.126975 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.126898    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:20:17.127123 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.127024    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bztmp\""
Oct 31 01:20:17.127173 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.127159    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:20:17.127215 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.127171    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-whdn9\""
Oct 31 01:20:17.128843 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.128828    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:20:17.131111 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-cert\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:17.131156 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-data-volume\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.131188 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131160    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-tmp-dir\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.131238 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.131283 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.131316 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-config-volume\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.131347 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrh8c\" (UniqueName: \"kubernetes.io/projected/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-kube-api-access-wrh8c\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.131377 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w8wh\" (UniqueName: \"kubernetes.io/projected/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-api-access-7w8wh\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.131407 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxd8\" (UniqueName: \"kubernetes.io/projected/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-kube-api-access-psxd8\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:17.131457 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131445    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-metrics-tls\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.131490 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.131464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-crio-socket\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.232434 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-metrics-tls\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.232434 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-crio-socket\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.232434 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-cert\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:17.232434 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-data-volume\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.232754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-tmp-dir\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.232754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.232754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.232754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232608    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-crio-socket\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.232754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-config-volume\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.232754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrh8c\" (UniqueName: \"kubernetes.io/projected/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-kube-api-access-wrh8c\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.232754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232709    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8wh\" (UniqueName: \"kubernetes.io/projected/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-api-access-7w8wh\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.232754 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxd8\" (UniqueName: \"kubernetes.io/projected/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-kube-api-access-psxd8\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:17.233155 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.232835    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-data-volume\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.233210 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.233195    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.233512 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.233469    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-tmp-dir\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.233512 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.233505    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-config-volume\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.236106 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.236085    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.236205 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.236087    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-metrics-tls\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.236300 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.236279    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-cert\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:17.246188 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.246163    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8wh\" (UniqueName: \"kubernetes.io/projected/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-api-access-7w8wh\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.246256 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.246239    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxd8\" (UniqueName: \"kubernetes.io/projected/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-kube-api-access-psxd8\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:17.246339 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.246324    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrh8c\" (UniqueName: \"kubernetes.io/projected/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-kube-api-access-wrh8c\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.333460 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.333421    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:17.342683 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.342669    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:17.352949 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.352930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:17.626347 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.626279    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lxzrw"]
Oct 31 01:20:17.629197 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.629172    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vs2gb"]
Oct 31 01:20:17.629933 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:17.629902    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc3aecc_79e1_42d2_ab4c_a4d7fcf8d70a.slice/crio-0497b1414424bdb3fd5bdf788a83b5e85c626f6571afc2535d5ebb91cb93e8fc WatchSource:0}: Error finding container 0497b1414424bdb3fd5bdf788a83b5e85c626f6571afc2535d5ebb91cb93e8fc: Status 404 returned error can't find the container with id 0497b1414424bdb3fd5bdf788a83b5e85c626f6571afc2535d5ebb91cb93e8fc
Oct 31 01:20:17.632121 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:17.632072    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8a7935_f8d1_4a05_ab04_08b62b0a9f99.slice/crio-1e9d84f281a645194d9babe72fd3b7d863ee8edde544cccf178632ad2b31e812 WatchSource:0}: Error finding container 1e9d84f281a645194d9babe72fd3b7d863ee8edde544cccf178632ad2b31e812: Status 404 returned error can't find the container with id 1e9d84f281a645194d9babe72fd3b7d863ee8edde544cccf178632ad2b31e812
Oct 31 01:20:17.657827 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.657808    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jtnln"]
Oct 31 01:20:17.659286 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:17.659264    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cb3e38_466f_4fab_8281_1257ca8fc8ae.slice/crio-7e70aa6674059cce777d2bca3379a049624888a91fd859b30cd4aed5fb5dbfea WatchSource:0}: Error finding container 7e70aa6674059cce777d2bca3379a049624888a91fd859b30cd4aed5fb5dbfea: Status 404 returned error can't find the container with id 7e70aa6674059cce777d2bca3379a049624888a91fd859b30cd4aed5fb5dbfea
Oct 31 01:20:17.675234 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.675209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:17.675326 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.675245    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:17.675379 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.675353    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:17.680120 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.680098    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:20:17.680216 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.680160    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:20:17.682393 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.682374    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z69qs\""
Oct 31 01:20:17.682483 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.682425    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:20:17.682483 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.682438    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbv59\""
Oct 31 01:20:17.682483 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.682457    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25cw\""
Oct 31 01:20:17.682625 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.682500    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:20:17.953738 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.953698    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtnln" event={"ID":"a3cb3e38-466f-4fab-8281-1257ca8fc8ae","Type":"ContainerStarted","Data":"7e70aa6674059cce777d2bca3379a049624888a91fd859b30cd4aed5fb5dbfea"}
Oct 31 01:20:17.955160 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.955135    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vs2gb" event={"ID":"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99","Type":"ContainerStarted","Data":"236a67c59daf7bf9f78ec5fedcd078c7240bd4ff9d4a36a99c050976c703a391"}
Oct 31 01:20:17.955523 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.955182    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vs2gb" event={"ID":"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99","Type":"ContainerStarted","Data":"1e9d84f281a645194d9babe72fd3b7d863ee8edde544cccf178632ad2b31e812"}
Oct 31 01:20:17.956290 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:17.956269    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lxzrw" event={"ID":"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a","Type":"ContainerStarted","Data":"0497b1414424bdb3fd5bdf788a83b5e85c626f6571afc2535d5ebb91cb93e8fc"}
Oct 31 01:20:18.960321 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:18.960288    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vs2gb" event={"ID":"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99","Type":"ContainerStarted","Data":"4d177b8f8695c362ad0fee983ae306b9e21dfecf50ef9d3dcc56235ea99a2a08"}
Oct 31 01:20:22.975998 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:22.975372    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vs2gb" event={"ID":"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99","Type":"ContainerStarted","Data":"6067cf9c82f397a040229c75a91a84bf983b320465c2cf66af56c12b6b5050a6"}
Oct 31 01:20:23.001556 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:23.001511    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vs2gb" podStartSLOduration=2.290373804 podStartE2EDuration="7.001497833s" totalImagesPullingTime="4.711124029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:20:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:23.001247022 +0000 UTC m=+52.839370914" watchObservedRunningTime="2026-10-31 01:20:23.001497833 +0000 UTC m=+52.839621720"
Oct 31 01:20:23.979174 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:23.979137    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtnln" event={"ID":"a3cb3e38-466f-4fab-8281-1257ca8fc8ae","Type":"ContainerStarted","Data":"5504b686cd9ed482e5e776c060c3e5a985ff817e70e00c99121c9757f53b1f2b"}
Oct 31 01:20:23.979635 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:23.979195    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtnln" event={"ID":"a3cb3e38-466f-4fab-8281-1257ca8fc8ae","Type":"ContainerStarted","Data":"1bfb508af9740a1d32fac32d227556a31164fcd9109b2a26fdb2ebd5b000bf40"}
Oct 31 01:20:23.979635 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:23.979222    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:23.980640 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:23.980620    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lxzrw" event={"ID":"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a","Type":"ContainerStarted","Data":"7b68eabf13b638eb7aab22fa09a44776a79768ad8f9598670b646617ea1cdc6c"}
Oct 31 01:20:24.001711 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.001668    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jtnln" podStartSLOduration=1.677998986 podStartE2EDuration="7.001656118s" totalImagesPullingTime="5.323657132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:24.001399822 +0000 UTC m=+53.839523715" watchObservedRunningTime="2026-10-31 01:20:24.001656118 +0000 UTC m=+53.839780008"
Oct 31 01:20:24.025162 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.025122    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lxzrw" podStartSLOduration=2.548745398 podStartE2EDuration="8.025111958s" totalImagesPullingTime="5.47636656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:24.024621478 +0000 UTC m=+53.862745367" watchObservedRunningTime="2026-10-31 01:20:24.025111958 +0000 UTC m=+53.863235847"
Oct 31 01:20:24.674566 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.674531    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"]
Oct 31 01:20:24.677815 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.677794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.681225 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.681204    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"]
Oct 31 01:20:24.684242 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.684226    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.689326 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.689309    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:20:24.689616 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.689595    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:20:24.689710 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.689649    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:20:24.689899 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.689880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npsk2\" (UniqueName: \"kubernetes.io/projected/dd5f332c-3b57-4a47-9dae-f191abaa5d52-kube-api-access-npsk2\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.689983 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.689915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwcpn\" (UniqueName: \"kubernetes.io/projected/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-api-access-cwcpn\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.689983 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.689946    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.690111 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.690164 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.690216 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.690268 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690214    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.690268 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.690363 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690314    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd5f332c-3b57-4a47-9dae-f191abaa5d52-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.690363 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.690616 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690596    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8cc5t"]
Oct 31 01:20:24.691026 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690944    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 01:20:24.691026 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.690984    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 01:20:24.691916 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.691895    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qvf5d\""
Oct 31 01:20:24.693767 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.693750    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"]
Oct 31 01:20:24.693850 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.693832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.697527 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.697463    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"]
Oct 31 01:20:24.735902 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.735869    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:20:24.736072 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.735972    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 01:20:24.736072 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.736011    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:20:24.736072 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.736028    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9rv2t\""
Oct 31 01:20:24.736177 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.736078    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-44nrr\""
Oct 31 01:20:24.747448 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.747431    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 01:20:24.787613 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.787591    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:20:24.787803 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.787788    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 01:20:24.791152 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791125    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.791232 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcpn\" (UniqueName: \"kubernetes.io/projected/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-api-access-cwcpn\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.791232 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-wtmp\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.791232 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-sys\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.791232 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-metrics-client-ca\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.791357 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791234    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.791357 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791302    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.791420 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.791420 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791379    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-tls\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.791420 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:24.791396    2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Oct 31 01:20:24.791420 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.791597 ip-10-0-130-49 kubenswrapper[2574]: E1031 01:20:24.791451    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls podName:d11ae2b3-41ce-4c4f-9e94-0fb9330700be nodeName:}" failed. No retries permitted until 2026-10-31 01:20:25.291431356 +0000 UTC m=+55.129555243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls") pod "kube-state-metrics-5dc8d7d89b-hxnfz" (UID: "d11ae2b3-41ce-4c4f-9e94-0fb9330700be") : secret "kube-state-metrics-tls" not found
Oct 31 01:20:24.791597 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791469    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsk2\" (UniqueName: \"kubernetes.io/projected/dd5f332c-3b57-4a47-9dae-f191abaa5d52-kube-api-access-npsk2\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.791709 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.791709 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-textfile\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.791709 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.791843 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-root\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.791895 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.791943 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.791927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2x5w\" (UniqueName: \"kubernetes.io/projected/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-kube-api-access-j2x5w\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.792023 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.792005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd5f332c-3b57-4a47-9dae-f191abaa5d52-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.792023 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.792015    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.792167 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.792072    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.792325 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.792302    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.792402 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.792310    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.792744 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.792685    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd5f332c-3b57-4a47-9dae-f191abaa5d52-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.793982 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.793957    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.794087 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.794072    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.794424 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.794409    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.806084 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.806067    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcpn\" (UniqueName: \"kubernetes.io/projected/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-api-access-cwcpn\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:24.806263 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.806247    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsk2\" (UniqueName: \"kubernetes.io/projected/dd5f332c-3b57-4a47-9dae-f191abaa5d52-kube-api-access-npsk2\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:24.893227 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-tls\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893241    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-textfile\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-root\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2x5w\" (UniqueName: \"kubernetes.io/projected/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-kube-api-access-j2x5w\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-wtmp\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-sys\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893397 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-metrics-client-ca\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893788 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893405    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-root\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893788 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893464    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-sys\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893788 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893557    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-wtmp\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.893788 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.893713    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-textfile\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.894054 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.894018    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.894121 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.894104    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-metrics-client-ca\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.895589 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.895566    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-tls\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.895815 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.895800    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.906959 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.906935    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2x5w\" (UniqueName: \"kubernetes.io/projected/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-kube-api-access-j2x5w\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:24.993659 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:24.993592    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:25.010703 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.010685    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:25.021068 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:25.021032    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6a153_63a8_4b4a_9ca4_e3f1796eb34b.slice/crio-6d69e21238061fb9dcb6349b2a7d7ab0cc993ece7b4ed15e24a82f868233b9b1 WatchSource:0}: Error finding container 6d69e21238061fb9dcb6349b2a7d7ab0cc993ece7b4ed15e24a82f868233b9b1: Status 404 returned error can't find the container with id 6d69e21238061fb9dcb6349b2a7d7ab0cc993ece7b4ed15e24a82f868233b9b1
Oct 31 01:20:25.269646 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.269519    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"]
Oct 31 01:20:25.272005 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:25.271979    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5f332c_3b57_4a47_9dae_f191abaa5d52.slice/crio-af746ba05f0790de823e9581abd37e5e45af6838ebe850a1a0e646cbbe179c30 WatchSource:0}: Error finding container af746ba05f0790de823e9581abd37e5e45af6838ebe850a1a0e646cbbe179c30: Status 404 returned error can't find the container with id af746ba05f0790de823e9581abd37e5e45af6838ebe850a1a0e646cbbe179c30
Oct 31 01:20:25.296463 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.296439    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:25.298805 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.298782    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:25.303680 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.303657    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:25.581481 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.581287    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"]
Oct 31 01:20:25.583868 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:25.583846    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11ae2b3_41ce_4c4f_9e94_0fb9330700be.slice/crio-4371ad719fd27dc6a8c281c06b02cc1b0ecba4c9414ef8c9cef9cb6a8018b572 WatchSource:0}: Error finding container 4371ad719fd27dc6a8c281c06b02cc1b0ecba4c9414ef8c9cef9cb6a8018b572: Status 404 returned error can't find the container with id 4371ad719fd27dc6a8c281c06b02cc1b0ecba4c9414ef8c9cef9cb6a8018b572
Oct 31 01:20:25.987841 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.987807    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8cc5t" event={"ID":"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b","Type":"ContainerStarted","Data":"6d69e21238061fb9dcb6349b2a7d7ab0cc993ece7b4ed15e24a82f868233b9b1"}
Oct 31 01:20:25.989683 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.989656    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd" event={"ID":"dd5f332c-3b57-4a47-9dae-f191abaa5d52","Type":"ContainerStarted","Data":"db1f18095a1c09ac3191aef4f23128f88f9e4d89980e88f048b2f5620e43623a"}
Oct 31 01:20:25.989817 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.989714    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd" event={"ID":"dd5f332c-3b57-4a47-9dae-f191abaa5d52","Type":"ContainerStarted","Data":"80c32e6c2efafae09c37ab5881182c8a593e0a561c75fe0ad7a2fa380f91100d"}
Oct 31 01:20:25.989817 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.989751    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd" event={"ID":"dd5f332c-3b57-4a47-9dae-f191abaa5d52","Type":"ContainerStarted","Data":"af746ba05f0790de823e9581abd37e5e45af6838ebe850a1a0e646cbbe179c30"}
Oct 31 01:20:25.990757 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:25.990736    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz" event={"ID":"d11ae2b3-41ce-4c4f-9e94-0fb9330700be","Type":"ContainerStarted","Data":"4371ad719fd27dc6a8c281c06b02cc1b0ecba4c9414ef8c9cef9cb6a8018b572"}
Oct 31 01:20:26.996117 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:26.996079    2574 generic.go:356] "Generic (PLEG): container finished" podID="88d6a153-63a8-4b4a-9ca4-e3f1796eb34b" containerID="80e81e2f24c4ceb20a741cde841be6e2635100d9da6bec39544520ffeb8d2e9d" exitCode=0
Oct 31 01:20:26.996565 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:26.996139    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8cc5t" event={"ID":"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b","Type":"ContainerDied","Data":"80e81e2f24c4ceb20a741cde841be6e2635100d9da6bec39544520ffeb8d2e9d"}
Oct 31 01:20:28.002468 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.002433    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8cc5t" event={"ID":"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b","Type":"ContainerStarted","Data":"b7c801d97bb9828ebf6f493b82b9e93687dcb33050e59b4e1856da6532a0e035"}
Oct 31 01:20:28.002904 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.002503    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8cc5t" event={"ID":"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b","Type":"ContainerStarted","Data":"4ed258d3e636ca127210bd2ef911b228459fcf6814750726f73b242d394c353d"}
Oct 31 01:20:28.004562 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.004538    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd" event={"ID":"dd5f332c-3b57-4a47-9dae-f191abaa5d52","Type":"ContainerStarted","Data":"6066e8e5353e58d423f1b4a2fe492702bea4b352e46d51f74ece0d5d8b8f0b99"}
Oct 31 01:20:28.006623 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.006594    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz" event={"ID":"d11ae2b3-41ce-4c4f-9e94-0fb9330700be","Type":"ContainerStarted","Data":"835d1273392d96f887bb9c016f9bb0e5e65512801df009c004240b80428a0aab"}
Oct 31 01:20:28.006694 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.006683    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz" event={"ID":"d11ae2b3-41ce-4c4f-9e94-0fb9330700be","Type":"ContainerStarted","Data":"916094e58d9cebd3ee0914e2abf1dd28babf90ae7d5f67e94417b938a9e3cfcf"}
Oct 31 01:20:28.006726 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.006709    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz" event={"ID":"d11ae2b3-41ce-4c4f-9e94-0fb9330700be","Type":"ContainerStarted","Data":"d82c1df618de70c30a6d5ac11ef1f1404d1fb0efdbac65c14b8f16c6c232eda7"}
Oct 31 01:20:28.026544 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.026504    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8cc5t" podStartSLOduration=2.735090958 podStartE2EDuration="4.026493672s" totalImagesPullingTime="1.216795126s" totalInitContainerRuntime="74.607588ms" isStatefulPod=true podCreationTimestamp="2026-10-31 01:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:28.025806347 +0000 UTC m=+57.863930239" watchObservedRunningTime="2026-10-31 01:20:28.026493672 +0000 UTC m=+57.864617563"
Oct 31 01:20:28.048924 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.048879    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd" podStartSLOduration=1.8702037740000002 podStartE2EDuration="4.048862499s" totalImagesPullingTime="2.178658725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:28.04856875 +0000 UTC m=+57.886692643" watchObservedRunningTime="2026-10-31 01:20:28.048862499 +0000 UTC m=+57.886986389"
Oct 31 01:20:28.076027 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.075989    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz" podStartSLOduration=2.124788171 podStartE2EDuration="4.075978466s" totalImagesPullingTime="1.951190295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:28.075300215 +0000 UTC m=+57.913424106" watchObservedRunningTime="2026-10-31 01:20:28.075978466 +0000 UTC m=+57.914102356"
Oct 31 01:20:28.924564 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:28.924538    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:29.124542 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.124507    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f555ffffc-q4tx9"]
Oct 31 01:20:29.127635 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.127620    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.139306 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.139278    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 01:20:29.139413 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.139328    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 01:20:29.140252 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.140233    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f555ffffc-q4tx9"]
Oct 31 01:20:29.160315 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.160298    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 01:20:29.160315 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.160310    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fk7jl\""
Oct 31 01:20:29.160315 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.160298    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 01:20:29.160486 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.160327    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6imjehf5hm31v\""
Oct 31 01:20:29.224825 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.224751    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.224825 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.224799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-metrics-server-audit-profiles\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.224985 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.224894    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-tls\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.224985 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.224913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfg8\" (UniqueName: \"kubernetes.io/projected/e435a839-2e87-46cf-a521-17e5ffa2dfc0-kube-api-access-bcfg8\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.224985 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.224931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e435a839-2e87-46cf-a521-17e5ffa2dfc0-audit-log\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.224985 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.224947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-client-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.225154 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.225031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-client-certs\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.326145 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.326115    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.326223 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.326160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-metrics-server-audit-profiles\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.326267 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.326221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-tls\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.326267 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.326245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfg8\" (UniqueName: \"kubernetes.io/projected/e435a839-2e87-46cf-a521-17e5ffa2dfc0-kube-api-access-bcfg8\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.326352 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.326268    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e435a839-2e87-46cf-a521-17e5ffa2dfc0-audit-log\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.326352 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.326292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-client-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.326352 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.326324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-client-certs\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.326757 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.326716    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e435a839-2e87-46cf-a521-17e5ffa2dfc0-audit-log\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.327064 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.327009    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.327248 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.327228    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-metrics-server-audit-profiles\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.329054 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.329021    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-client-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.329128 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.329097    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-client-certs\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.329166 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.329144    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-tls\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.340000 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.339982    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfg8\" (UniqueName: \"kubernetes.io/projected/e435a839-2e87-46cf-a521-17e5ffa2dfc0-kube-api-access-bcfg8\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.438945 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.438915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:29.488192 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.488168    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"]
Oct 31 01:20:29.498546 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.498520    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:29.508594 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.507962    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"]
Oct 31 01:20:29.508594 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.508183    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vdjwv\""
Oct 31 01:20:29.508594 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.508265    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 01:20:29.528473 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.528449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5cgz\" (UniqueName: \"kubernetes.io/projected/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-kube-api-access-d5cgz\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:29.528589 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.528560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:29.628853 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.628822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5cgz\" (UniqueName: \"kubernetes.io/projected/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-kube-api-access-d5cgz\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:29.628986 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.628906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:29.631457 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.631435    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:29.642367 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.642341    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5cgz\" (UniqueName: \"kubernetes.io/projected/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-kube-api-access-d5cgz\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:29.714554 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.714517    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f555ffffc-q4tx9"]
Oct 31 01:20:29.716908 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:29.716888    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode435a839_2e87_46cf_a521_17e5ffa2dfc0.slice/crio-cbe72075f6340937182b29b9b8f16cc2c9c05604cb83ad8e74869d71991015a1 WatchSource:0}: Error finding container cbe72075f6340937182b29b9b8f16cc2c9c05604cb83ad8e74869d71991015a1: Status 404 returned error can't find the container with id cbe72075f6340937182b29b9b8f16cc2c9c05604cb83ad8e74869d71991015a1
Oct 31 01:20:29.813951 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:29.813869    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:30.013455 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:30.013419    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9" event={"ID":"e435a839-2e87-46cf-a521-17e5ffa2dfc0","Type":"ContainerStarted","Data":"cbe72075f6340937182b29b9b8f16cc2c9c05604cb83ad8e74869d71991015a1"}
Oct 31 01:20:30.086499 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:30.086377    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"]
Oct 31 01:20:30.088437 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:30.088406    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e1fa2a_b9a6_479e_a0ab_ad118dd4c47b.slice/crio-64289842f588128f5e6a376b39d53e4db125553015523734f752c430a662e850 WatchSource:0}: Error finding container 64289842f588128f5e6a376b39d53e4db125553015523734f752c430a662e850: Status 404 returned error can't find the container with id 64289842f588128f5e6a376b39d53e4db125553015523734f752c430a662e850
Oct 31 01:20:31.020353 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:31.020315    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp" event={"ID":"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b","Type":"ContainerStarted","Data":"64289842f588128f5e6a376b39d53e4db125553015523734f752c430a662e850"}
Oct 31 01:20:32.024851 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:32.024750    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9" event={"ID":"e435a839-2e87-46cf-a521-17e5ffa2dfc0","Type":"ContainerStarted","Data":"7e2141d4de0af91b7fc978143681aee678b7f9316c77eb4e48f02cd7db85b226"}
Oct 31 01:20:32.026302 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:32.026270    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp" event={"ID":"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b","Type":"ContainerStarted","Data":"df42268c5aac85b54375f0005c5027088336257a077a7cd3fa7bfd8bab0db88a"}
Oct 31 01:20:32.026414 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:32.026399    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:32.031293 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:32.031273    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:32.051499 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:32.051460    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9" podStartSLOduration=1.364356804 podStartE2EDuration="3.051448193s" totalImagesPullingTime="1.687091389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:32.050710916 +0000 UTC m=+61.888834806" watchObservedRunningTime="2026-10-31 01:20:32.051448193 +0000 UTC m=+61.889572083"
Oct 31 01:20:32.072716 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:32.072680    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp" podStartSLOduration=1.390051925 podStartE2EDuration="3.072669723s" totalImagesPullingTime="1.682617798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:32.072324781 +0000 UTC m=+61.910448666" watchObservedRunningTime="2026-10-31 01:20:32.072669723 +0000 UTC m=+61.910793640"
Oct 31 01:20:33.985393 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:33.985364    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:36.390162 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.390128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:36.394246 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.394230    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:20:36.403551 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.403531    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:36.490523 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.490492    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:36.490684 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.490534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:36.494485 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.494470    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:20:36.494704 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.494689    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:20:36.502943 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.502924    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:36.504172 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.504160    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:20:36.513695 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.513672    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:36.593901 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.593867    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbv59\""
Oct 31 01:20:36.596638 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.596625    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25cw\""
Oct 31 01:20:36.600804 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.600789    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:36.601218 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.601207    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z69qs\""
Oct 31 01:20:36.603323 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.603311    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:36.608004 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.607990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:36.883130 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.883100    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-67sdw"]
Oct 31 01:20:36.887361 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:36.887332    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0826e21_1b23_4f42_a0af_eb1621a858f5.slice/crio-a4cf745efb9cc5292f2902105eb1eae24ec796230b9493883f9eac451213ee54 WatchSource:0}: Error finding container a4cf745efb9cc5292f2902105eb1eae24ec796230b9493883f9eac451213ee54: Status 404 returned error can't find the container with id a4cf745efb9cc5292f2902105eb1eae24ec796230b9493883f9eac451213ee54
Oct 31 01:20:36.897685 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.897661    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4drdm"]
Oct 31 01:20:36.900014 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:36.899988    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d44c81_99b9_4df8_982a_98edaa068b3d.slice/crio-23d6f8e35e5a5839bb6859526998fb4c53f3b88d552fab79be70f698adffb73c WatchSource:0}: Error finding container 23d6f8e35e5a5839bb6859526998fb4c53f3b88d552fab79be70f698adffb73c: Status 404 returned error can't find the container with id 23d6f8e35e5a5839bb6859526998fb4c53f3b88d552fab79be70f698adffb73c
Oct 31 01:20:36.909936 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:36.909919    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sgnh6"]
Oct 31 01:20:36.911983 ip-10-0-130-49 kubenswrapper[2574]: W1031 01:20:36.911965    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552bfc4c_f3a9_4b18_a7d2_4f7e3d8aa2d6.slice/crio-234db5284f1fe0f6393ca25d95779c8d706ad502fdec6ab0e814e13a7998983a WatchSource:0}: Error finding container 234db5284f1fe0f6393ca25d95779c8d706ad502fdec6ab0e814e13a7998983a: Status 404 returned error can't find the container with id 234db5284f1fe0f6393ca25d95779c8d706ad502fdec6ab0e814e13a7998983a
Oct 31 01:20:37.043890 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:37.043852    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-67sdw" event={"ID":"e0826e21-1b23-4f42-a0af-eb1621a858f5","Type":"ContainerStarted","Data":"a4cf745efb9cc5292f2902105eb1eae24ec796230b9493883f9eac451213ee54"}
Oct 31 01:20:37.045078 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:37.045055    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4drdm" event={"ID":"71d44c81-99b9-4df8-982a-98edaa068b3d","Type":"ContainerStarted","Data":"23d6f8e35e5a5839bb6859526998fb4c53f3b88d552fab79be70f698adffb73c"}
Oct 31 01:20:37.046159 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:37.046136    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sgnh6" event={"ID":"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6","Type":"ContainerStarted","Data":"234db5284f1fe0f6393ca25d95779c8d706ad502fdec6ab0e814e13a7998983a"}
Oct 31 01:20:39.056636 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:39.056594    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4drdm" event={"ID":"71d44c81-99b9-4df8-982a-98edaa068b3d","Type":"ContainerStarted","Data":"0666504e6b071f5c906455a93d3b473413f35f1edcc3b0a53de6c205eb1d765d"}
Oct 31 01:20:39.057138 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:39.056667    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4drdm" event={"ID":"71d44c81-99b9-4df8-982a-98edaa068b3d","Type":"ContainerStarted","Data":"9ee1360b54386a494d898568e538b48c4fe4af17eec195a9b291bcb7061d357f"}
Oct 31 01:20:39.078890 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:39.078827    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4drdm" podStartSLOduration=66.849343015 podStartE2EDuration="1m8.078808819s" totalImagesPullingTime="1.229465804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:39.078786202 +0000 UTC m=+68.916910093" watchObservedRunningTime="2026-10-31 01:20:39.078808819 +0000 UTC m=+68.916932708"
Oct 31 01:20:41.066157 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:41.066121    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-67sdw" event={"ID":"e0826e21-1b23-4f42-a0af-eb1621a858f5","Type":"ContainerStarted","Data":"318b58c58c6dc80648b9489cda6c44d672d8aa03266c2624ecdda075c5a46c13"}
Oct 31 01:20:41.066903 ip-10-0-130-49 kubenswrapper[2574]: I1031 01:20:41.066879    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:41.979266 ip-10-0-130-49 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 01:20:41.987820 ip-10-0-130-49 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 01:20:41.987996 ip-10-0-130-49 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 01:20:41.988063 ip-10-0-130-49 systemd[1]: kubelet.service: Consumed 3.743s CPU time.
Oct 31 01:20:42.010498 ip-10-0-130-49 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:20:42.080941 ip-10-0-130-49 kubenswrapper[5293]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:20:42.080941 ip-10-0-130-49 kubenswrapper[5293]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:20:42.080941 ip-10-0-130-49 kubenswrapper[5293]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:20:42.080941 ip-10-0-130-49 kubenswrapper[5293]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:20:42.083147 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083069    5293 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:42.083147 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083146    5293 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083151    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083154    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083157    5293 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083160    5293 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083163    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083165    5293 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083176    5293 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083180    5293 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083183    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083186    5293 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083189    5293 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083191    5293 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083194    5293 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083196    5293 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083199    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083202    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083204    5293 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083207    5293 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083209    5293 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:42.083221 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083212    5293 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083214    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083217    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083220    5293 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083222    5293 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083225    5293 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083231    5293 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083235    5293 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083237    5293 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083240    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083243    5293 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083245    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083248    5293 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083251    5293 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083253    5293 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083256    5293 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083258    5293 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083261    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083263    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:42.083721 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083266    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083268    5293 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083271    5293 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083273    5293 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083276    5293 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083278    5293 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083281    5293 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083283    5293 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083287    5293 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083289    5293 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083292    5293 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083295    5293 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083298    5293 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083300    5293 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083303    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083306    5293 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083308    5293 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083311    5293 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083313    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:42.084212 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083316    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083319    5293 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083323    5293 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083327    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083330    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083332    5293 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083335    5293 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083337    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083340    5293 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083342    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083345    5293 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083348    5293 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083350    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083353    5293 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083355    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083358    5293 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083360    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083363    5293 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083365    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:42.084682 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083369    5293 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083372    5293 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083375    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083377    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083380    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083383    5293 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083385    5293 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083388    5293 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083391    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083393    5293 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083396    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083398    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083401    5293 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083404    5293 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083406    5293 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083409    5293 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083411    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083414    5293 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083417    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:42.085172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083419    5293 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083422    5293 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083425    5293 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083428    5293 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083430    5293 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083433    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083435    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083438    5293 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.083440    5293 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084210    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084217    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084220    5293 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084223    5293 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084225    5293 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084230    5293 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084233    5293 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084236    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084238    5293 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084242    5293 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084244    5293 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:42.085647 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084247    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084249    5293 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084253    5293 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084255    5293 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084258    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084262    5293 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084266    5293 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084269    5293 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084272    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084275    5293 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084278    5293 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084280    5293 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084283    5293 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084286    5293 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084288    5293 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084291    5293 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084294    5293 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084296    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084299    5293 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:42.086172 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084301    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084304    5293 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084308    5293 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084326    5293 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084331    5293 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084335    5293 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084339    5293 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084342    5293 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084345    5293 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084348    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084351    5293 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084354    5293 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084356    5293 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084359    5293 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084361    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084364    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084367    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084369    5293 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084372    5293 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084375    5293 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:42.086638 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084377    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084380    5293 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084383    5293 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084385    5293 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084388    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084391    5293 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084393    5293 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084396    5293 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084399    5293 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084401    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084404    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084407    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084410    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084413    5293 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084415    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084418    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084420    5293 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084423    5293 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084426    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:42.087139 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084428    5293 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084431    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084434    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084436    5293 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084439    5293 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084441    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084444    5293 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084446    5293 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084449    5293 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084451    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084454    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084457    5293 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084459    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084462    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084464    5293 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084467    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084469    5293 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084472    5293 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084475    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:42.087598 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084478    5293 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084480    5293 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084483    5293 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084485    5293 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084488    5293 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084491    5293 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084493    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084496    5293 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084499    5293 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084501    5293 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084503    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084506    5293 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084511    5293 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084515    5293 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084518    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084521    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084524    5293 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.084527    5293 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085033    5293 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085054    5293 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:42.088315 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085057    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085062    5293 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085065    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085069    5293 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085072    5293 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085075    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085078    5293 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085080    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085083    5293 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085085    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085088    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085091    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085094    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085096    5293 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085099    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085102    5293 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085105    5293 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085108    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085110    5293 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:42.088928 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085113    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085116    5293 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085118    5293 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085121    5293 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085123    5293 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085126    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085128    5293 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085131    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085133    5293 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085136    5293 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085139    5293 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085141    5293 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085144    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085146    5293 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085149    5293 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085151    5293 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085158    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085161    5293 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085163    5293 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085166    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:42.089583 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085168    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085171    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085175    5293 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085179    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085183    5293 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085186    5293 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085189    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085192    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085195    5293 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085197    5293 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085200    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085203    5293 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085205    5293 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085208    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085210    5293 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085213    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085215    5293 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085218    5293 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085220    5293 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085223    5293 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:42.090115 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085225    5293 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085228    5293 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085231    5293 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085233    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085236    5293 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085239    5293 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085241    5293 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085243    5293 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085247    5293 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085249    5293 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085252    5293 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085255    5293 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085257    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085260    5293 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085263    5293 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085266    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085268    5293 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085271    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085273    5293 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085276    5293 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:42.090607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085278    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085281    5293 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085283    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085286    5293 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085288    5293 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085291    5293 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085293    5293 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085296    5293 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085298    5293 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085301    5293 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085303    5293 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085306    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085309    5293 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085311    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085313    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085316    5293 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085320    5293 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085322    5293 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085325    5293 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:42.091114 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085328    5293 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085330    5293 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085333    5293 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085336    5293 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085338    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.085341    5293 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085434    5293 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085456    5293 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085465    5293 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085472    5293 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085477    5293 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085480    5293 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085485    5293 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085489    5293 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085492    5293 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085495    5293 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085499    5293 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085502    5293 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085505    5293 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085509    5293 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085511    5293 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085514    5293 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085517    5293 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085521    5293 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:20:42.091572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085526    5293 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085528    5293 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085531    5293 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085535    5293 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085538    5293 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085543    5293 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085546    5293 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085549    5293 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085552    5293 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085555    5293 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085558    5293 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085561    5293 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085564    5293 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085567    5293 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085571    5293 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085574    5293 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085577    5293 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085580    5293 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085583    5293 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085586    5293 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085590    5293 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085593    5293 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085596    5293 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085599    5293 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:20:42.092230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085602    5293 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085606    5293 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085609    5293 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085612    5293 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085615    5293 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085618    5293 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085620    5293 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085623    5293 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085626    5293 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085629    5293 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085631    5293 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085634    5293 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085639    5293 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085643    5293 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085647    5293 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085650    5293 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085654    5293 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085657    5293 flags.go:64] FLAG: --help="false"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085660    5293 flags.go:64] FLAG: --hostname-override="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085663    5293 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085666    5293 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085668    5293 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085672    5293 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:20:42.093117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085676    5293 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085679    5293 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085682    5293 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085685    5293 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085687    5293 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085690    5293 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085694    5293 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085696    5293 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085699    5293 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085703    5293 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085706    5293 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085708    5293 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085711    5293 flags.go:64] FLAG: --lock-file=""
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085714    5293 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085717    5293 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085720    5293 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085725    5293 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085728    5293 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085731    5293 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085734    5293 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085737    5293 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085741    5293 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085746    5293 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085748    5293 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085753    5293 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:20:42.093907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085764    5293 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085769    5293 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085772    5293 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085775    5293 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085778    5293 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085780    5293 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085783    5293 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085788    5293 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085790    5293 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085798    5293 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085801    5293 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085804    5293 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085807    5293 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085810    5293 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085813    5293 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085816    5293 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085819    5293 flags.go:64] FLAG: --port="10250"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085822    5293 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085825    5293 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0515dd301b49ade2b"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085828    5293 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085831    5293 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085834    5293 flags.go:64] FLAG: --register-node="true"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085837    5293 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085841    5293 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085844    5293 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085846    5293 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:20:42.094558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085849    5293 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085853    5293 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085857    5293 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085861    5293 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085864    5293 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085867    5293 flags.go:64] FLAG: --runonce="false"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085870    5293 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085874    5293 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085877    5293 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085879    5293 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085882    5293 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085885    5293 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085888    5293 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085893    5293 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085896    5293 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085898    5293 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085901    5293 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085904    5293 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085907    5293 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085910    5293 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085913    5293 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085918    5293 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085921    5293 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085924    5293 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085928    5293 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:20:42.095206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085931    5293 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085934    5293 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085937    5293 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085940    5293 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085943    5293 flags.go:64] FLAG: --v="2"
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085947    5293 flags.go:64] FLAG: --version="false"
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085951    5293 flags.go:64] FLAG: --vmodule=""
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085955    5293 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.085958    5293 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086102    5293 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086107    5293 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086110    5293 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086113    5293 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086115    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086118    5293 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086121    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086124    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086126    5293 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086129    5293 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086131    5293 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086135    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086138    5293 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:42.095792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086141    5293 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086143    5293 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086146    5293 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086149    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086152    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086154    5293 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086157    5293 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086160    5293 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086162    5293 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086165    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086168    5293 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086170    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086173    5293 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086176    5293 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086180    5293 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086184    5293 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086187    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086190    5293 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086193    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086196    5293 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:42.096382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086200    5293 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086202    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086205    5293 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086208    5293 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086210    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086213    5293 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086216    5293 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086219    5293 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086221    5293 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086224    5293 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086227    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086230    5293 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086233    5293 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086235    5293 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086238    5293 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086240    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086243    5293 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086246    5293 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086249    5293 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086251    5293 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:42.097001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086254    5293 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086256    5293 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086259    5293 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086262    5293 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086266    5293 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086270    5293 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086273    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086276    5293 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086279    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086282    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086284    5293 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086287    5293 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086291    5293 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086293    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086296    5293 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086298    5293 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086301    5293 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086303    5293 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086306    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:42.097766 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086309    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086311    5293 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086314    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086318    5293 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086321    5293 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086323    5293 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086326    5293 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086329    5293 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086331    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086334    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086336    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086339    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086342    5293 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086344    5293 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086347    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086349    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086352    5293 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086355    5293 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086357    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:42.098266 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086360    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086362    5293 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086365    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086367    5293 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086370    5293 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086373    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086376    5293 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086379    5293 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086381    5293 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086384    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086386    5293 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086389    5293 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086391    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086394    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086397    5293 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:42.098740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.086423    5293 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086458    5293 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086463    5293 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086466    5293 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086469    5293 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086471    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086474    5293 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086477    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086479    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086482    5293 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086484    5293 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086487    5293 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086490    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086492    5293 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086495    5293 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086497    5293 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086500    5293 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086502    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086505    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086507    5293 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:42.099236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086510    5293 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086513    5293 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086515    5293 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086519    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086521    5293 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086524    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086527    5293 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086529    5293 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086533    5293 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086537    5293 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086540    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086543    5293 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086546    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086548    5293 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086552    5293 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086555    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086557    5293 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086560    5293 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086563    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:42.099998 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086566    5293 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086568    5293 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086571    5293 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086573    5293 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086576    5293 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086578    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086581    5293 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086583    5293 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086586    5293 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086588    5293 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086591    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086594    5293 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086596    5293 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086598    5293 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086601    5293 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086604    5293 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086608    5293 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086611    5293 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086614    5293 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086618    5293 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:42.100558 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086621    5293 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086624    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086627    5293 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086629    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086632    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086635    5293 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086638    5293 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086642    5293 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086646    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086648    5293 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086651    5293 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086654    5293 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086656    5293 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086659    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086661    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086664    5293 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086666    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086669    5293 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086671    5293 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:42.101083 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086674    5293 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086676    5293 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086679    5293 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086682    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086684    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086686    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086690    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086692    5293 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086695    5293 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086699    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086702    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086704    5293 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086707    5293 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086710    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086712    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086715    5293 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086717    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086720    5293 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086722    5293 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:42.101534 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086725    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086729    5293 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086731    5293 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086734    5293 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086736    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086739    5293 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086742    5293 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086744    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086747    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.086749    5293 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.091102    5293 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.091116    5293 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.091134    5293 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.091139    5293 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091195    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091201    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091205    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091207    5293 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091210    5293 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:42.101986 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091213    5293 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091215    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091218    5293 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091221    5293 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091223    5293 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091226    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091229    5293 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091231    5293 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091234    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091236    5293 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091239    5293 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091241    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091244    5293 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091247    5293 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091250    5293 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091252    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091256    5293 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091259    5293 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091261    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091264    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:42.102693 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091266    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091269    5293 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091271    5293 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091274    5293 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091276    5293 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091279    5293 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091282    5293 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091284    5293 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091286    5293 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091290    5293 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091294    5293 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091299    5293 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091303    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091306    5293 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091309    5293 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091312    5293 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091314    5293 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091317    5293 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091319    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:42.103400 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091322    5293 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091324    5293 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091327    5293 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091330    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091332    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091334    5293 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091337    5293 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091340    5293 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091343    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091346    5293 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091349    5293 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091352    5293 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091354    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091357    5293 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091359    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091361    5293 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091364    5293 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091367    5293 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091369    5293 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:42.104216 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091371    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091374    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091376    5293 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091379    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091381    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091384    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091386    5293 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091389    5293 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091391    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091394    5293 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091396    5293 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091399    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091401    5293 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091404    5293 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091407    5293 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091409    5293 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091411    5293 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091414    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091431    5293 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:42.105032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091434    5293 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091437    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091441    5293 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091444    5293 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091447    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091449    5293 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091452    5293 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091454    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091457    5293 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091459    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091462    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091464    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091467    5293 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091470    5293 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091473    5293 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091475    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091478    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091480    5293 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091483    5293 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091485    5293 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:42.105874 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091488    5293 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091490    5293 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091493    5293 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091495    5293 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.091522    5293 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091649    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091653    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091656    5293 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091659    5293 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091661    5293 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091664    5293 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091668    5293 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091671    5293 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091674    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091677    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091684    5293 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:42.106792 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091687    5293 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091689    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091692    5293 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091694    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091697    5293 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091699    5293 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091702    5293 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091705    5293 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091708    5293 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091710    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091713    5293 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091716    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091718    5293 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091721    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091723    5293 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091725    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091728    5293 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091731    5293 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091733    5293 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091736    5293 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:42.107541 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091738    5293 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091741    5293 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091743    5293 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091746    5293 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091748    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091750    5293 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091753    5293 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091755    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091758    5293 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091760    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091763    5293 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091766    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091771    5293 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091774    5293 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091776    5293 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091779    5293 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091781    5293 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091784    5293 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091787    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091789    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:42.108468 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091791    5293 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091794    5293 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091796    5293 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091799    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091801    5293 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091804    5293 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091806    5293 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091809    5293 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091811    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091814    5293 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091816    5293 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091819    5293 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091822    5293 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091824    5293 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091826    5293 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091830    5293 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091832    5293 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091834    5293 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091837    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:42.109392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091839    5293 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091842    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091844    5293 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091846    5293 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091849    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091851    5293 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091854    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091857    5293 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091860    5293 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091862    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091864    5293 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091867    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091870    5293 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091872    5293 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091875    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091877    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091880    5293 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091882    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091885    5293 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091887    5293 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:42.110401 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091890    5293 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091892    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091895    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091897    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091899    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091902    5293 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091904    5293 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091907    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091909    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091912    5293 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091916    5293 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091920    5293 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091923    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091925    5293 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091928    5293 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091930    5293 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:42.111227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.091953    5293 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091982    5293 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091986    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091989    5293 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091992    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091994    5293 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091997    5293 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.091999    5293 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092001    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092004    5293 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092006    5293 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092009    5293 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092011    5293 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092014    5293 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092016    5293 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092019    5293 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092021    5293 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092023    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092026    5293 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:42.111900 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092028    5293 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092031    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092033    5293 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092052    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092057    5293 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092061    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092064    5293 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092066    5293 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092070    5293 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092072    5293 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092075    5293 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092078    5293 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092080    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092082    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092087    5293 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092090    5293 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092092    5293 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092095    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092098    5293 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092100    5293 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:42.112569 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092103    5293 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092105    5293 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092108    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092110    5293 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092113    5293 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092116    5293 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092119    5293 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092121    5293 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092124    5293 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092126    5293 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092129    5293 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092131    5293 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092133    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092136    5293 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092139    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092141    5293 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092144    5293 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092146    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092149    5293 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:42.113055 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092151    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092154    5293 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092156    5293 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092158    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092161    5293 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092164    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092166    5293 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092171    5293 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092174    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092177    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092179    5293 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092182    5293 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092186    5293 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092188    5293 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092191    5293 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092194    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092196    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092199    5293 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092201    5293 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092204    5293 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:42.113518 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092207    5293 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092214    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092216    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092219    5293 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092222    5293 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092226    5293 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092229    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092232    5293 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092235    5293 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092238    5293 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092241    5293 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092243    5293 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092246    5293 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092249    5293 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092251    5293 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092254    5293 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092258    5293 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092261    5293 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092264    5293 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:42.114026 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092268    5293 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092271    5293 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092273    5293 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092276    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092278    5293 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092281    5293 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092284    5293 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092286    5293 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092290    5293 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:20:42.092293    5293 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.092341    5293 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.094216    5293 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.094280    5293 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.094482    5293 server.go:1014] "Starting client certificate rotation"
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.094621    5293 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.094676    5293 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 01:14:31 +0000 UTC" deadline="2028-04-21 05:22:29.727143947 +0000 UTC"
Oct 31 01:20:42.114623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.094720    5293 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12916h1m47.632427052s"
Oct 31 01:20:42.115320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.094987    5293 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:20:42.115320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.096001    5293 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:20:42.115320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.097179    5293 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:20:42.115320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.098900    5293 log.go:25] "Validated CRI v1 image API"
Oct 31 01:20:42.115320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.099622    5293 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:20:42.115320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.101499    5293 fs.go:135] Filesystem UUIDs: map[1836a17a-b10e-46e9-b6f5-0e264596ab75:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cd2541a1-10e3-4f23-964c-0265e0f797d9:/dev/nvme0n1p3]
Oct 31 01:20:42.115631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.101517    5293 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02e96a9b79e9dba006fbe6548382d8dfb674f7620a2085b17b620875713f2df0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02e96a9b79e9dba006fbe6548382d8dfb674f7620a2085b17b620875713f2df0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0497b1414424bdb3fd5bdf788a83b5e85c626f6571afc2535d5ebb91cb93e8fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0497b1414424bdb3fd5bdf788a83b5e85c626f6571afc2535d5ebb91cb93e8fc/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b92ed9766035f2143dc143b1cd43b5a568dba8b24897e79a0907c03a5f07103/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b92ed9766035f2143dc143b1cd43b5a568dba8b24897e79a0907c03a5f07103/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d4693ff8bd909d8ccb79431d239f4dce367d4180e7db58b9a30cb8a127fa23c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d4693ff8bd909d8ccb79431d239f4dce367d4180e7db58b9a30cb8a127fa23c/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e9d84f281a645194d9babe72fd3b7d863ee8edde544cccf178632ad2b31e812/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e9d84f281a645194d9babe72fd3b7d863ee8edde544cccf178632ad2b31e812/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/234db5284f1fe0f6393ca25d95779c8d706ad502fdec6ab0e814e13a7998983a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/234db5284f1fe0f6393ca25d95779c8d706ad502fdec6ab0e814e13a7998983a/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23d6f8e35e5a5839bb6859526998fb4c53f3b88d552fab79be70f698adffb73c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23d6f8e35e5a5839bb6859526998fb4c53f3b88d552fab79be70f698adffb73c/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3920eea7457abf4cbc916b6eb03a38f78a18061dc99e826ab98fe57b259bb478/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3920eea7457abf4cbc916b6eb03a38f78a18061dc99e826ab98fe57b259bb478/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4371ad719fd27dc6a8c281c06b02cc1b0ecba4c9414ef8c9cef9cb6a8018b572/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4371ad719fd27dc6a8c281c06b02cc1b0ecba4c9414ef8c9cef9cb6a8018b572/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58405bcd9e73457678a0eac63a61fabfd983ac65db7e3adbb7c1b8c6ff2369e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58405bcd9e73457678a0eac63a61fabfd983ac65db7e3adbb7c1b8c6ff2369e3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/635ac6929582ced57b84c752fb07c0f1892859ad65d4be1dc595d16533337047/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/635ac6929582ced57b84c752fb07c0f1892859ad65d4be1dc595d16533337047/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64289842f588128f5e6a376b39d53e4db125553015523734f752c430a662e850/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64289842f588128f5e6a376b39d53e4db125553015523734f752c430a662e850/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d69e21238061fb9dcb6349b2a7d7ab0cc993ece7b4ed15e24a82f868233b9b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d69e21238061fb9dcb6349b2a7d7ab0cc993ece7b4ed15e24a82f868233b9b1/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e70aa6674059cce777d2bca3379a049624888a91fd859b30cd4aed5fb5dbfea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e70aa6674059cce777d2bca3379a049624888a91fd859b30cd4aed5fb5dbfea/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91a120bc27513dade25956dde8909d486d7728987442391f9ddf58171ba7a6db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91a120bc27513dade25956dde8909d486d7728987442391f9ddf58171ba7a6db/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d90f2c4d59c098e522caa5577fedc67736a0a5eab95555956da3583f8c39655/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d90f2c4d59c098e522caa5577fedc67736a0a5eab95555956da3583f8c39655/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4cf745efb9cc5292f2902105eb1eae24ec796230b9493883f9eac451213ee54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4cf745efb9cc5292f2902105eb1eae24ec796230b9493883f9eac451213ee54/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af746ba05f0790de823e9581abd37e5e45af6838ebe850a1a0e646cbbe179c30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af746ba05f0790de823e9581abd37e5e45af6838ebe850a1a0e646cbbe179c30/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd0dd9ba261dee9861e3af7fb84f797f612bf9475cd15d09ca0eba4d78077789/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd0dd9ba261dee9861e3af7fb84f797f612bf9475cd15d09ca0eba4d78077789/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbe72075f6340937182b29b9b8f16cc2c9c05604cb83ad8e74869d71991015a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbe72075f6340937182b29b9b8f16cc2c9c05604cb83ad8e74869d71991015a1/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dac0236c1f9a8755121e5ddf0aad3212802c1f6e8ac830ba13e4e22d27288aa9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dac0236c1f9a8755121e5ddf0aad3212802c1f6e8ac830ba13e4e22d27288aa9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b074486-2077-4e57-8223-f51a2bf0bea3/volumes/kubernetes.io~projected/kube-api-access-rskls:{mountpoint:/var/lib/kubelet/pods/0b074486-2077-4e57-8223-f51a2bf0bea3/volumes/kubernetes.io~projected/kube-api-access-rskls major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~projected/kube-api-access-zp9bh:{mountpoint:/var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~projected/kube-api-access-zp9bh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/471cd70e-c62e-4f48-903d-1d71fa617d35/volumes/kubernetes.io~projected/kube-api-access-2b85m:{mountpoint:/var/lib/kubelet/pods/471cd70e-c62e-4f48-903d-1d71fa617d35/volumes/kubernetes.io~projected/kube-api-access-2b85m major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volumes/kubernetes.io~projected/kube-api-access-7swnn:{mountpoint:/var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volumes/kubernetes.io~projected/kube-api-access-7swnn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4950850b-9fc3-4c93-a3c8-b92ef5d906cc/volumes/kubernetes.io~projected/kube-api-access-dmn4x:{mountpoint:/var/lib/kubelet/pods/4950850b-9fc3-4c93-a3c8-b92ef5d906cc/volumes/kubernetes.io~projected/kube-api-access-dmn4x major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b391665-5177-4a9c-adfb-d8d9d2772ab9/volumes/kubernetes.io~projected/kube-api-access-58jmj:{mountpoint:/var/lib/kubelet/pods/5b391665-5177-4a9c-adfb-d8d9d2772ab9/volumes/kubernetes.io~projected/kube-api-access-58jmj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a/volumes/kubernetes.io~projected/kube-api-access-psxd8:{mountpoint:/var/lib/kubelet/pods/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a/volumes/kubernetes.io~projected/kube-api-access-psxd8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71d44c81-99b9-4df8-982a-98edaa068b3d/volumes/kubernetes.io~projected/kube-api-access-f499c:{mountpoint:/var/lib/kubelet/pods/71d44c81-99b9-4df8-982a-98edaa068b3d/volumes/kubernetes.io~projected/kube-api-access-f499c major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71d44c81-99b9-4df8-982a-98edaa068b3d/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/71d44c81-99b9-4df8-982a-98edaa068b3d/volumes/kubernetes.io~secret/metrics-certs major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406/volumes/kubernetes.io~projected/kube-api-access-tnw7z:{mountpoint:/var/lib/kubelet/pods/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406/volumes/kubernetes.io~projected/kube-api-access-tnw7z major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f8bac4d-65fb-4079-ad54-a762105a2d93/volumes/kubernetes.io~projected/kube-api-access-s2bcz:{mountpoint:/var/lib/kubelet/pods/7f8bac4d-65fb-4079-ad54-a762105a2d93/volumes/kubernetes.io~projected/kube-api-access-s2bcz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~projected/kube-api-access-j2x5w:{mountpoint:/var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~projected/kube-api-access-j2x5w major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b/volumes/kubernetes.io~projected/kube-api-access-d5cgz:{mountpoint:/var/lib/kubelet/pods/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b/volumes/kubernetes.io~projected/kube-api-access-d5cgz major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3cb3e38-466f-4fab-8281-1257ca8fc8ae/volumes/kubernetes.io~projected/kube-api-access-wrh8c:{mountpoint:/var/lib/kubelet/pods/a3cb3e38-466f-4fab-8281-1257ca8fc8ae/volumes/kubernetes.io~projected/kube-api-access-wrh8c major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3cb3e38-466f-4fab-8281-1257ca8fc8ae/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/a3cb3e38-466f-4fab-8281-1257ca8fc8ae/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~projected/kube-api-access-cwcpn:{mountpoint:/var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~projected/kube-api-access-cwcpn major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~projected/kube-api-access-npsk2:{mountpoint:/var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~projected/kube-api-access-npsk2 major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0826e21-1b23-4f42-a0af-eb1621a858f5/volumes/kubernetes.io~projected/kube-api-access-59gbm:{mountpoint:/var/lib/kubelet/pods/e0826e21-1b23-4f42-a0af-eb1621a858f5/volumes/kubernetes.io~projected/kube-api-access-59gbm major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~projected/kube-api-access-bcfg8:{mountpoint:/var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~projected/kube-api-access-bcfg8 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99/volumes/kubernetes.io~projected/kube-api-access-7w8wh:{mountpoint:/var/lib/kubelet/pods/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99/volumes/kubernetes.io~projected/kube-api-access-7w8wh major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/01a097181a695e79554f91bc309bacae1d9f6616d5a5e94611d7c26c1576bf3d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5dc3d015df7ff3978642b1f383a2aab988a6106ce6170af750f5b7e644697478/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3ca1ef6e55f18b9cdf2baea04c5fc7ad1be4d0e2c5136814f49706fbd0134a1d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/e1dd5e6635a687c980cdfafab3406580d94f41e81dea0265ab019c3322f13ca9/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/590aa3976a13954d3097affd53eac8b2e483e66d544f5727486fb1d2d40183ff/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d752f362ce8be8f1a589f4b896c61c5aba0af4a43f5877a42de339ea90b751e1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/47ee7093f63bf9ce3705667cee2b34c92fee1d1ec20f627c83b62101a74ed1d8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5ea9fb3154cb37b893a58d7fbd40805e77c69cae8be8b54c51852bbc09a61801/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a91bffd31a0f390616f242ccdba27e4633fde413c271c4f935e844410507f2ef/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f59f1c9c7cfe21877306c1b9e29c052c3735267f35de8c498548d0c2ecddf8a6/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/e7f46f49524fb07425c3399434ec8911436debf8842d5994f1d88b54a5d42a55/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/caa8adbcc6f0a63720087c307eda8d699c372e7325e05dae40749a1ae48ee43a/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f5e044316c35551ceb3ec225525851a6247601cdb6a3b3b4ed6437b47c6e174e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/30302e5ae2c585c931c7690d2a609823c446d35634b4bc331cee8899cadbfee3/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e691194b035bf376931dfbbe917c85f6b7506ad3849c2c4d451232471a71ba01/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5c4eb9f0edfa3b6e516d8a7a8f91d4983b13d9571e4fa2ab873f31b78f6e541a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b87f4bd88c29654e64f7e80eecde2084379998102cd8c49dec291833949afe70/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/3cb755074b000eadcfe6c619e9b1878f8a58fdb6c14254883c605db27a83e509/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/a54e2531db2bfbf0a40c578f00f48fdbfbc963b6ef5c4eecfedc527fa63ca3d8/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cce4df7efb5c0527d88ec557e9f64e9a76cdf549b30cb4806f1636c32e09b5d7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/91dbda72358ac83563f008d84ef7b360dd0b661682859c0faaaa1a77b4f680a1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5ef15e98a3059be12d4d11a32881a870a4a5ab16d2ac6e03b2aecb4679b7af41/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ab675a92910ef9a8e2dc96b951096b4f76af6b8593e052c342f0bd02d40df162/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/07fc07264113969ee2b5066067b88a5390a798aac407c3a53151a2ebe0312c46/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/2abd81e16800d34d7d5326de5994b0b633321d99e847be0cbb5f78dbf33a2f81/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/ff7aa25940a5167389241c03be1d38a55ab33859063552296beb4e6caaecaaab/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/53e0582815be49ef6f4f9f555de278ca529ef056a8790f98b54ee2ccd4593c88/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/22daba5bfbb2509bba3bb04cc82d03075f60a393936068ea2889df1f27f9574d/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/dc2b8cec6201ea94141800ee10a1f1c67bebbaf9c2d7a9594d070c858b85fefb/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/a0365461bcee82cfe70bb7bd39fa9c7c134586bc312c459a3a0e7b0ca74150d3/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9ad143608fb8848e0c68d079439c725a0687b9c89a6fc8869b1f183b7f75b8a1/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/37180d195baa46be2153b92b5b31a2f85c78261404b95da02b3a5d3fac1619b2/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/13cf1ac10e066dddb6333054b69e38b1a3032a7183bb45dbc0858674ccfa147f/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/b36b0f0f32d57a837a5a60bfc1a493e60cd9c469e6254c499a464423145fbd6c/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/238d182ea07199f36ecf44e20cd3b9dcfd7b333c4f7e3e9827d5daff2140e6da/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/0106b53df680229827e03b5590deccb5289866b08b5e248ae7ef7ba6d546b174/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/710a349826eda471fa40be702908c1bfc538faea78cbe8272ce0f6eea59ef38c/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/fef3ca674aac19aa77acabc329c750de23e4bde065ca4249b0443b8e7263f21e/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/3a786148c3c4d52a3370ef8b900099e72a680b5ef48f8d235f2639f37bd2d805/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/5f1e5fd47ef3ffca05af7520a35ecef5ae392c35c9a35892d72d00a1d11f4202/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/1e56a51f8f9874f05916998ac903620c5de2aa2ea3e4229fdffae67f13b50746/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/1d8faa76bc6a79bb1373f4a8efc7ee62bee2b9bd38f7405f140342d99fff68fa/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/3475c2467c99b9e285a238ddb3a10155eb89485669e3b9d59acdd819d70098ee/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/341c9dc623f2ac0ca14885658cb04e86b79a350af0c9307eeb85d5820fdd5004/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/332e3200b231d0903df321cb81ff6df1c8e1b760d3c7f86a3dab65d2e3e3a25c/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/897c1ec922feed2bbde8dbdc3becba1c0535c89bf74533f715d66373accab567/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/7e7073c22b1f3f7b32e7c610593c2c5d82503fad0d73e0a8db14d098278e5e31/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/357a8c79261e158d8e4c42abd063ddd00faa5d7b2bf02915ada0f82558cacede/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/9cc03a38a4610416da586902426b45d22fda1bad9c86dfff829888111f69eeb6/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/77912cbfdb5cfddd3b79da8c30026125b66cd36dc5a89efdea5e41636e6531c4/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/8a0bd71e0e8eb0796b1efadbed69a1ee932f43eed45c375895f1702c9d348fe7/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/4511288a9b4bc8239e9310dfaa245bcb313c60b72bcbe20068036d6e8b749895/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/80d8bedd58689ac8911f7b8a92a22fdbbb9e1b809cf104489378b0ae82425172/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e7bd9797cfef9536469144897632f22a685a7c9f6929e210cc50dacf733c725d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fe1ba2f6ee683b28eafdbb64c7f773ca587f183a57658004cc0b28c8ccf704c6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e331efec68db20184b478b2c5b66b1eedea3c2c59682b57153a78ee40d90f37b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6932286db28f23a154bd2b8bb4b3d878948753fb9bf650b98cb0144ffed01f05/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b5019e2c70473e649adc81c3af010bbd333ea57a5719f2433fdf283e67a461fe/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/f4d8f8ee7da4fa87bef719bec6aa9ef5e06e9b75024f718dc9e30076259adfc4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/a53437b0263ef5f90ce5895a57c2b938db793443ac6090d32572a5129eae25a2/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e64861cce96549579b5375b05fa7472b9384a9040958ddb030ab90c802d31dc3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/545202ff6a9fdfc5e6cf9c6d5404712a43c44035635259637422d4d031775d41/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.111082    5293 manager.go:217] Machine: {Timestamp:2026-10-31 01:20:42.11028589 +0000 UTC m=+0.058934274 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094664 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec241ee80ca73c3fbef4aa5d15275a6b SystemUUID:ec241ee8-0ca7-3c3f-bef4-aa5d15275a6b BootID:eb47b728-9a73-45df-aaf5-b0c768ed0019 Filesystems:[{Device:/run/containers/storage/overlay-containers/02e96a9b79e9dba006fbe6548382d8dfb674f7620a2085b17b620875713f2df0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3920eea7457abf4cbc916b6eb03a38f78a18061dc99e826ab98fe57b259bb478/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406/volumes/kubernetes.io~projected/kube-api-access-tnw7z DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64289842f588128f5e6a376b39d53e4db125553015523734f752c430a662e850/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:319 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b074486-2077-4e57-8223-f51a2bf0bea3/volumes/kubernetes.io~projected/kube-api-access-rskls DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volumes/kubernetes.io~projected/kube-api-access-7swnn DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3cb3e38-466f-4fab-8281-1257ca8fc8ae/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0497b1414424bdb3fd5bdf788a83b5e85c626f6571afc2535d5ebb91cb93e8fc/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e70aa6674059cce777d2bca3379a049624888a91fd859b30cd4aed5fb5dbfea/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af746ba05f0790de823e9581abd37e5e45af6838ebe850a1a0e646cbbe179c30/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~projected/kube-api-access-bcfg8 DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b/volumes/kubernetes.io~projected/kube-api-access-d5cgz DeviceMajor:0 DeviceMinor:320 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d69e21238061fb9dcb6349b2a7d7ab0cc993ece7b4ed15e24a82f868233b9b1/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a/volumes/kubernetes.io~projected/kube-api-access-psxd8 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3cb3e38-466f-4fab-8281-1257ca8fc8ae/volumes/kubernetes.io~projected/kube-api-access-wrh8c DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91a120bc27513dade25956dde8909d486d7728987442391f9ddf58171ba7a6db/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71d44c81-99b9-4df8-982a-98edaa068b3d/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d90f2c4d59c098e522caa5577fedc67736a0a5eab95555956da3583f8c39655/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b92ed9766035f2143dc143b1cd43b5a568dba8b24897e79a0907c03a5f07103/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4cf745efb9cc5292f2902105eb1eae24ec796230b9493883f9eac451213ee54/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~projected/kube-api-access-cwcpn DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58405bcd9e73457678a0eac63a61fabfd983ac65db7e3adbb7c1b8c6ff2369e3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71d44c81-99b9-4df8-982a-98edaa068b3d/volumes/kubernetes.io~projected/kube-api-access-f499c DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4371ad719fd27dc6a8c281c06b02cc1b0ecba4c9414ef8c9cef9cb6a8018b572/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~projected/kube-api-access-zp9bh DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23d6f8e35e5a5839bb6859526998fb4c53f3b88d552fab79be70f698adffb73c/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4950850b-9fc3-4c93-a3c8-b92ef5d906cc/volumes/kubernetes.io~projected/kube-api-access-dmn4x DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e435a839-2e87-46cf-a521-17e5ffa2dfc0/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d11ae2b3-41ce-4c4f-9e94-0fb9330700be/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:268 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbe72075f6340937182b29b9b8f16cc2c9c05604cb83ad8e74869d71991015a1/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/635ac6929582ced57b84c752fb07c0f1892859ad65d4be1dc595d16533337047/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5b391665-5177-4a9c-adfb-d8d9d2772ab9/volumes/kubernetes.io~projected/kube-api-access-58jmj DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3fd62759-3b38-496d-b853-9d7bde9eb09d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/234db5284f1fe0f6393ca25d95779c8d706ad502fdec6ab0e814e13a7998983a/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d4693ff8bd909d8ccb79431d239f4dce367d4180e7db58b9a30cb8a127fa23c/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd5f332c-3b57-4a47-9dae-f191abaa5d52/volumes/kubernetes.io~projected/kube-api-access-npsk2 DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f8bac4d-65fb-4079-ad54-a762105a2d93/volumes/kubernetes.io~projected/kube-api-access-s2bcz DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99/volumes/kubernetes.io~projected/kube-api-access-7w8wh DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~projected/kube-api-access-j2x5w DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0826e21-1b23-4f42-a0af-eb1621a858f5/volumes/kubernetes.io~projected/kube-api-access-59gbm DeviceMajor:0 DeviceMinor:341 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dac0236c1f9a8755121e5ddf0aad3212802c1f6e8ac830ba13e4e22d27288aa9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:255 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/471cd70e-c62e-4f48-903d-1d71fa617d35/volumes/kubernetes.io~projected/kube-api-access-2b85m DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/4749d5cb-885e-418c-82ff-55ce8029e532/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e9d84f281a645194d9babe72fd3b7d863ee8edde544cccf178632ad2b31e812/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd0dd9ba261dee9861e3af7fb84f797f612bf9475cd15d09ca0eba4d78077789/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0497b1414424bdb MacAddress:fa:5b:96:39:6b:5f Speed:10000 Mtu:8901} {Name:1e9d84f281a6451 MacAddress:4a:c0:42:ba:d1:9a Speed:10000 Mtu:8901} {Name:234db5284f1fe0f MacAddress:5a:be:b3:c5:37:9c Speed:10000 Mtu:8901} {Name:23d6f8e35e5a583 MacAddress:fe:60:9c:2a:81:01 Speed:10000 Mtu:8901} {Name:4371ad719fd27dc MacAddress:1a:5b:3b:a8:4d:e5 Speed:10000 Mtu:8901} {Name:64289842f588128 MacAddress:ee:72:59:47:9a:57 Speed:10000 Mtu:8901} {Name:7e70aa6674059cc MacAddress:7e:6e:97:e2:f1:72 Speed:10000 Mtu:8901} {Name:a4cf745efb9cc52 MacAddress:e2:8f:e2:b3:f5:b0 Speed:10000 Mtu:8901} {Name:af746ba05f0790d MacAddress:d2:ad:87:2c:6f:f9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ea:6c:3a:9b Speed:0 Mtu:9001} {Name:br-int MacAddress:62:69:8f:8c:91:6f Speed:0 Mtu:8901} {Name:cbe72075f634093 MacAddress:1e:f8:09:20:bf:40 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ea:6c:3a:9b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:47:2f:7c:1e:b7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ba:14:c4:72:84:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.112113    5293 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.112184    5293 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.112674    5293 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.112712    5293 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-49.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113109    5293 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113117    5293 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113132    5293 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113152    5293 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113239    5293 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113382    5293 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113474    5293 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113509    5293 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113522    5293 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113532    5293 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113540    5293 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.113548    5293 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.114137    5293 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.114147    5293 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:20:42.115966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.114562    5293 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:20:42.117767 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.117744    5293 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:20:42.117767 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.117766    5293 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:20:42.117892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.117883    5293 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 01:20:42.118125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118110    5293 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:20:42.118190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118150    5293 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:20:42.118273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118262    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:20:42.118335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118281    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:20:42.118335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118291    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:20:42.118335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118300    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:20:42.118335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118308    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:20:42.118335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118316    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:20:42.118335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118325    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:20:42.118335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118334    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:20:42.118636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118345    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:20:42.118636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118356    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:20:42.118636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118380    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:20:42.118636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118394    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:20:42.118636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118422    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:20:42.118636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.118431    5293 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:20:42.121809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.121792    5293 server.go:1263] "Started kubelet"
Oct 31 01:20:42.121899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.121865    5293 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:20:42.122350 ip-10-0-130-49 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:20:42.123806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.123789    5293 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:20:42.123985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.123925    5293 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:20:42.124117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.124015    5293 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:20:42.126097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.124998    5293 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:20:42.127545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.127514    5293 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:20:42.127838 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.127721    5293 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:20:42.127915 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.127867    5293 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:33 +0000 UTC" deadline="2028-03-31 06:30:42.956353306 +0000 UTC"
Oct 31 01:20:42.127915 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.127907    5293 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12413h10m0.828449795s"
Oct 31 01:20:42.128543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.128137    5293 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:20:42.128543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.128152    5293 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:20:42.128543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.128263    5293 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:20:42.131034 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:20:42.129814    5293 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-49.us-east-2.compute.internal\" not found"
Oct 31 01:20:42.131034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.130026    5293 factory.go:55] Registering systemd factory
Oct 31 01:20:42.131034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.130067    5293 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:20:42.131034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.130550    5293 factory.go:153] Registering CRI-O factory
Oct 31 01:20:42.131034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.130563    5293 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:20:42.131034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.130615    5293 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:20:42.131034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.130633    5293 factory.go:103] Registering Raw factory
Oct 31 01:20:42.131034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.130650    5293 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:20:42.132614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.131971    5293 manager.go:319] Starting recovery of all containers
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133461    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71d44c81-99b9-4df8-982a-98edaa068b3d" volumeName="kubernetes.io/projected/71d44c81-99b9-4df8-982a-98edaa068b3d-kube-api-access-f499c" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133508    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f8bac4d-65fb-4079-ad54-a762105a2d93" volumeName="kubernetes.io/projected/7f8bac4d-65fb-4079-ad54-a762105a2d93-kube-api-access-s2bcz" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133527    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e435a839-2e87-46cf-a521-17e5ffa2dfc0" volumeName="kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133543    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4749d5cb-885e-418c-82ff-55ce8029e532" volumeName="kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-config" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133558    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4749d5cb-885e-418c-82ff-55ce8029e532" volumeName="kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-env-overrides" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133569    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0826e21-1b23-4f42-a0af-eb1621a858f5" volumeName="kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133581    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea8a7935-f8d1-4a05-ab04-08b62b0a9f99" volumeName="kubernetes.io/secret/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133592    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471cd70e-c62e-4f48-903d-1d71fa617d35" volumeName="kubernetes.io/projected/471cd70e-c62e-4f48-903d-1d71fa617d35-kube-api-access-2b85m" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133606    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b391665-5177-4a9c-adfb-d8d9d2772ab9" volumeName="kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-daemon-config" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133618    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d6a153-63a8-4b4a-9ca4-e3f1796eb34b" volumeName="kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133641    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d6a153-63a8-4b4a-9ca4-e3f1796eb34b" volumeName="kubernetes.io/projected/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-kube-api-access-j2x5w" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133653    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d6a153-63a8-4b4a-9ca4-e3f1796eb34b" volumeName="kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133665    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e435a839-2e87-46cf-a521-17e5ffa2dfc0" volumeName="kubernetes.io/projected/e435a839-2e87-46cf-a521-17e5ffa2dfc0-kube-api-access-bcfg8" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133678    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e435a839-2e87-46cf-a521-17e5ffa2dfc0" volumeName="kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133695    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea8a7935-f8d1-4a05-ab04-08b62b0a9f99" volumeName="kubernetes.io/projected/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-api-access-7w8wh" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133708    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406" volumeName="kubernetes.io/projected/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kube-api-access-tnw7z" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133720    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cb3e38-466f-4fab-8281-1257ca8fc8ae" volumeName="kubernetes.io/projected/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-kube-api-access-wrh8c" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133737    5293 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133744    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d11ae2b3-41ce-4c4f-9e94-0fb9330700be" volumeName="kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133764    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e435a839-2e87-46cf-a521-17e5ffa2dfc0" volumeName="kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-client-ca-bundle" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133776    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea8a7935-f8d1-4a05-ab04-08b62b0a9f99" volumeName="kubernetes.io/configmap/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133787    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4749d5cb-885e-418c-82ff-55ce8029e532" volumeName="kubernetes.io/secret/4749d5cb-885e-418c-82ff-55ce8029e532-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133799    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd62759-3b38-496d-b853-9d7bde9eb09d" volumeName="kubernetes.io/projected/3fd62759-3b38-496d-b853-9d7bde9eb09d-kube-api-access-zp9bh" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133810    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b391665-5177-4a9c-adfb-d8d9d2772ab9" volumeName="kubernetes.io/projected/5b391665-5177-4a9c-adfb-d8d9d2772ab9-kube-api-access-58jmj" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133822    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a" volumeName="kubernetes.io/projected/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-kube-api-access-psxd8" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133832    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f8bac4d-65fb-4079-ad54-a762105a2d93" volumeName="kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133845    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd62759-3b38-496d-b853-9d7bde9eb09d" volumeName="kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-tmp" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133857    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471cd70e-c62e-4f48-903d-1d71fa617d35" volumeName="kubernetes.io/configmap/471cd70e-c62e-4f48-903d-1d71fa617d35-serviceca" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133869    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a" volumeName="kubernetes.io/secret/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-cert" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133886    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd5f332c-3b57-4a47-9dae-f191abaa5d52" volumeName="kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133901    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b074486-2077-4e57-8223-f51a2bf0bea3" volumeName="kubernetes.io/projected/0b074486-2077-4e57-8223-f51a2bf0bea3-kube-api-access-rskls" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133912    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4950850b-9fc3-4c93-a3c8-b92ef5d906cc" volumeName="kubernetes.io/projected/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-kube-api-access-dmn4x" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133923    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cb3e38-466f-4fab-8281-1257ca8fc8ae" volumeName="kubernetes.io/configmap/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-config-volume" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133934    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d11ae2b3-41ce-4c4f-9e94-0fb9330700be" volumeName="kubernetes.io/empty-dir/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-volume-directive-shadow" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133946    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d11ae2b3-41ce-4c4f-9e94-0fb9330700be" volumeName="kubernetes.io/projected/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-api-access-cwcpn" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133959    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d11ae2b3-41ce-4c4f-9e94-0fb9330700be" volumeName="kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133970    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e435a839-2e87-46cf-a521-17e5ffa2dfc0" volumeName="kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133982    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4950850b-9fc3-4c93-a3c8-b92ef5d906cc" volumeName="kubernetes.io/empty-dir/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-tmp-dir" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.133994    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4749d5cb-885e-418c-82ff-55ce8029e532" volumeName="kubernetes.io/projected/4749d5cb-885e-418c-82ff-55ce8029e532-kube-api-access-7swnn" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134018    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b" volumeName="kubernetes.io/secret/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-monitoring-plugin-cert" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134030    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb0eaa17-d3d8-4db5-b63c-1d74a7d77996" volumeName="kubernetes.io/configmap/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-konnectivity-ca" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134060    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb0eaa17-d3d8-4db5-b63c-1d74a7d77996" volumeName="kubernetes.io/secret/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-agent-certs" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134072    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b391665-5177-4a9c-adfb-d8d9d2772ab9" volumeName="kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134105    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71d44c81-99b9-4df8-982a-98edaa068b3d" volumeName="kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134116    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f8bac4d-65fb-4079-ad54-a762105a2d93" volumeName="kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134127    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cb3e38-466f-4fab-8281-1257ca8fc8ae" volumeName="kubernetes.io/secret/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-metrics-tls" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134142    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e435a839-2e87-46cf-a521-17e5ffa2dfc0" volumeName="kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134153    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b074486-2077-4e57-8223-f51a2bf0bea3" volumeName="kubernetes.io/configmap/0b074486-2077-4e57-8223-f51a2bf0bea3-iptables-alerter-script" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134164    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fd62759-3b38-496d-b853-9d7bde9eb09d" volumeName="kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-tuned" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134175    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d6a153-63a8-4b4a-9ca4-e3f1796eb34b" volumeName="kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-tls" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134186    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f8bac4d-65fb-4079-ad54-a762105a2d93" volumeName="kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134199    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d6a153-63a8-4b4a-9ca4-e3f1796eb34b" volumeName="kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134212    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b" volumeName="kubernetes.io/projected/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-kube-api-access-d5cgz" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134225    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e435a839-2e87-46cf-a521-17e5ffa2dfc0" volumeName="kubernetes.io/empty-dir/e435a839-2e87-46cf-a521-17e5ffa2dfc0-audit-log" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134236    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cb3e38-466f-4fab-8281-1257ca8fc8ae" volumeName="kubernetes.io/empty-dir/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-tmp-dir" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134247    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d11ae2b3-41ce-4c4f-9e94-0fb9330700be" volumeName="kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134258    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd5f332c-3b57-4a47-9dae-f191abaa5d52" volumeName="kubernetes.io/configmap/dd5f332c-3b57-4a47-9dae-f191abaa5d52-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134278    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd5f332c-3b57-4a47-9dae-f191abaa5d52" volumeName="kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134291    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4749d5cb-885e-418c-82ff-55ce8029e532" volumeName="kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134302    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d6a153-63a8-4b4a-9ca4-e3f1796eb34b" volumeName="kubernetes.io/empty-dir/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-textfile" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134313    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d11ae2b3-41ce-4c4f-9e94-0fb9330700be" volumeName="kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134324    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd5f332c-3b57-4a47-9dae-f191abaa5d52" volumeName="kubernetes.io/projected/dd5f332c-3b57-4a47-9dae-f191abaa5d52-kube-api-access-npsk2" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134339    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea8a7935-f8d1-4a05-ab04-08b62b0a9f99" volumeName="kubernetes.io/empty-dir/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-data-volume" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134350    5293 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6" volumeName="kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret" seLinuxMountContext=""
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134363    5293 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134370    5293 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.134655    5293 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.135922    5293 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:20:42.138784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.137096    5293 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 01:20:42.163982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.163929    5293 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:20:42.165545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.165518    5293 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:20:42.165545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.165544    5293 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:20:42.165718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.165569    5293 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:20:42.165718 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:20:42.165611    5293 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 01:20:42.172921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.172900    5293 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:20:42.181260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.181219    5293 generic.go:356] "Generic (PLEG): container finished" podID="88d6a153-63a8-4b4a-9ca4-e3f1796eb34b" containerID="80e81e2f24c4ceb20a741cde841be6e2635100d9da6bec39544520ffeb8d2e9d" exitCode=0
Oct 31 01:20:42.193657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.193638    5293 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="c90f3a0044ba0976f7c176e5740bf924b55fef80cd64bf5ff1a61c1039195077" exitCode=0
Oct 31 01:20:42.193657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.193655    5293 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="88a5364e0cb107bce7a3afde13e245007e2cb49b01092549c88f2cd5e1f3b3ea" exitCode=0
Oct 31 01:20:42.193657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.193660    5293 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="f814a66700a0ddfff84cc8627bc5ae57d493fe4b231288b66f19d0bb20babe65" exitCode=0
Oct 31 01:20:42.193845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.193665    5293 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="fdb21c5c81d6c49ba621b29af3cf2bc9dcde24267132a607c6425141bfdf277e" exitCode=0
Oct 31 01:20:42.193845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.193670    5293 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="e33292854a98931337733508f89c67c501d5ab2a75a3abc69d2a6b08dd10e7cc" exitCode=0
Oct 31 01:20:42.193845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.193679    5293 generic.go:356] "Generic (PLEG): container finished" podID="7f8bac4d-65fb-4079-ad54-a762105a2d93" containerID="5bd8dffaa6837b759185e2972417b4ae3d581f69e2da2f6fd81472658a7d88e9" exitCode=0
Oct 31 01:20:42.223649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.223618    5293 generic.go:356] "Generic (PLEG): container finished" podID="d36f9ad789e8b4baae6a17685ca5cf7f" containerID="82dd953d8c031bf46fef706237fae69f65cff0fa5be2272a97d9957016d12278" exitCode=0
Oct 31 01:20:42.230834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.230785    5293 manager.go:324] Recovery completed
Oct 31 01:20:42.252011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.251991    5293 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:20:42.252011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252006    5293 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:20:42.252197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252024    5293 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:20:42.252197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252188    5293 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 01:20:42.252302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252203    5293 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 01:20:42.252302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252220    5293 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 01:20:42.252302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252226    5293 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 01:20:42.252302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252231    5293 policy_none.go:50] "Start"
Oct 31 01:20:42.252302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252240    5293 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:20:42.252302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.252248    5293 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:20:42.253328 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.253314    5293 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 01:20:42.253362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.253329    5293 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 01:20:42.253362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.253335    5293 policy_none.go:44] "Start"
Oct 31 01:20:42.257704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.257666    5293 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:20:42.261285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.261269    5293 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:20:42.261981 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:20:42.261293    5293 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:20:42.261981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.261982    5293 server.go:87] "Starting device plugin registration server"
Oct 31 01:20:42.262253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.262241    5293 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:20:42.262307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.262253    5293 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:20:42.262351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.262331    5293 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:20:42.262462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.262414    5293 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:20:42.262462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.262448    5293 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:20:42.262462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.262454    5293 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:20:42.262679 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.262522    5293 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:20:42.26243754Z","UUID":"10e7be68-5819-48cb-acb3-d81d49fe2c0f","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:20:42.264008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.263988    5293 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:20:42.264008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.264017    5293 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:20:42.264202 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:20:42.264164    5293 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:20:42.266516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.265936    5293 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"]
Oct 31 01:20:42.267090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.266967    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"b3e5d96baa84771b6810c7d4e7a868fe","Type":"ContainerStarted","Data":"99273834aa319291a8a9f6a37a236859e7cc8012895d41a186c837087a7c41f2"}
Oct 31 01:20:42.267090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.267030    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"b3e5d96baa84771b6810c7d4e7a868fe","Type":"ContainerStarted","Data":"dac0236c1f9a8755121e5ddf0aad3212802c1f6e8ac830ba13e4e22d27288aa9"}
Oct 31 01:20:42.268162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.267370    5293 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.268385 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.267532    5293 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.268475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.268408    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"d36f9ad789e8b4baae6a17685ca5cf7f","Type":"ContainerStarted","Data":"5eb99b666d36a8cd580c29d16d540b7d3f9c335b416b9ece865a4de83d19d267"}
Oct 31 01:20:42.268571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.268547    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"d36f9ad789e8b4baae6a17685ca5cf7f","Type":"ContainerDied","Data":"82dd953d8c031bf46fef706237fae69f65cff0fa5be2272a97d9957016d12278"}
Oct 31 01:20:42.268688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.268667    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal" event={"ID":"d36f9ad789e8b4baae6a17685ca5cf7f","Type":"ContainerStarted","Data":"1d4693ff8bd909d8ccb79431d239f4dce367d4180e7db58b9a30cb8a127fa23c"}
Oct 31 01:20:42.282751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.282729    5293 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:20:42.282864 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:20:42.282769    5293 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.283348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.283331    5293 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:20:42.283418 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:20:42.283363    5293 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.338290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.338263    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.338290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.338294    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3e5d96baa84771b6810c7d4e7a868fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"b3e5d96baa84771b6810c7d4e7a868fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.338461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.338310    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.363387 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.363363    5293 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:20:42.365367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.365351    5293 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:20:42.365431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.365382    5293 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:20:42.365431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.365393    5293 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-49.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:20:42.365431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.365429    5293 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.377543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.377526    5293 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.377617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.377579    5293 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.438942 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.438913    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3e5d96baa84771b6810c7d4e7a868fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"b3e5d96baa84771b6810c7d4e7a868fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.439098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.438948    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.439098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.438968    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.439098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.439013    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.439098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.439032    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d36f9ad789e8b4baae6a17685ca5cf7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"d36f9ad789e8b4baae6a17685ca5cf7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:42.439098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:42.439034    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3e5d96baa84771b6810c7d4e7a868fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal\" (UID: \"b3e5d96baa84771b6810c7d4e7a868fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:43.113841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.113808    5293 apiserver.go:52] "Watching apiserver"
Oct 31 01:20:43.119915 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.119891    5293 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:20:43.120505 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.120485    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mg8qc","openshift-dns/dns-default-jtnln","openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz","openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp","openshift-dns/node-resolver-jp6vv","openshift-network-diagnostics/network-check-target-67sdw","openshift-cluster-node-tuning-operator/tuned-kl48l","openshift-multus/multus-gvxxk","kube-system/konnectivity-agent-c9rzr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d","openshift-multus/multus-additional-cni-plugins-ncbph","openshift-ovn-kubernetes/ovnkube-node-b2jz4","kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-fp4dj","openshift-monitoring/node-exporter-8cc5t","openshift-monitoring/metrics-server-5f555ffffc-q4tx9","openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd","openshift-multus/network-metrics-daemon-4drdm","kube-system/global-pull-secret-syncer-sgnh6","openshift-insights/insights-runtime-extractor-vs2gb","openshift-ingress-canary/ingress-canary-lxzrw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"]
Oct 31 01:20:43.125358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125330    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:20:43.125463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125388    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:20:43.125463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125399    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.125463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125463    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.125635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125527    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.125635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125570    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:20:43.125727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125659    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm2lp\""
Oct 31 01:20:43.125809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125787    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h7598\""
Oct 31 01:20:43.125980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125957    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:20:43.126057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.125958    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:20:43.126104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.126075    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:20:43.126347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.126327    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:20:43.126525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.126511    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.126607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.126593    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:20:43.127988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.127967    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z69qs\""
Oct 31 01:20:43.131807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.131789    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:20:43.131899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.131878    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:20:43.131966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.131906    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.132366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132346    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.132366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132341    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.132504 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132466    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.132504 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132502    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.132703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132680    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:20:43.132760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132704    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.132760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132709    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.132760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132716    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.132760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132711    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.132760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132688    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.132760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132688    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:20:43.132760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.132728    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.133095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.133071    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:20:43.133095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.133087    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.133223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.133080    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.133223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.133076    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.134252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134219    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.134252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134241    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.134252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134230    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:20:43.134380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134243    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:20:43.134468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134452    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:20:43.134595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134582    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 01:20:43.134672 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134660    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:20:43.134729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134714    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 01:20:43.134759 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.134733    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:20:43.135233 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.135223    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 01:20:43.140677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140660    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-hosts-file\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:20:43.140717 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140690    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-etc-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.140746 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140714    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-netns\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.140746 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140735    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-conf-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.140798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140783    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-systemd-units\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.140842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140829    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-modprobe-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.140873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140860    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-sys\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.140915 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140901    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:43.140944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140937    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-daemon-config\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.141013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.140996    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.141100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141085    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-ovn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.141185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141169    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-dbus\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:43.141219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141183    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-daemon-config\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.141258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141218    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b074486-2077-4e57-8223-f51a2bf0bea3-iptables-alerter-script\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:20:43.141258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141238    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.141258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141243    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jmj\" (UniqueName: \"kubernetes.io/projected/5b391665-5177-4a9c-adfb-d8d9d2772ab9-kube-api-access-58jmj\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.141345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141326    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-original-pull-secret\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:43.141374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141343    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.141374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141367    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:43.141429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141391    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b074486-2077-4e57-8223-f51a2bf0bea3-iptables-alerter-script\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:20:43.141468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141402    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.141926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141534    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-host\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.141926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141537    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.142004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141947    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.142004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.141973    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.142087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142001    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-config\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.142087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142026    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4749d5cb-885e-418c-82ff-55ce8029e532-ovn-node-metrics-cert\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.142443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142420    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysconfig\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.142503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142450    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-config\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.142503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142475    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.142598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142502    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-systemd\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.142598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142532    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-konnectivity-ca\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:20:43.142598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142564    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-socket-dir-parent\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.142734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142624    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-multus-certs\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.142734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142622    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4749d5cb-885e-418c-82ff-55ce8029e532-ovn-node-metrics-cert\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.142734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142693    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-socket-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.142876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142755    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-slash\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.142876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142781    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-konnectivity-ca\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:20:43.142876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142790    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-netns\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.142876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142854    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.143074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142881    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-run\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.143074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142931    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-systemd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.143074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142961    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-agent-certs\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:20:43.143074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.142998    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b074486-2077-4e57-8223-f51a2bf0bea3-host-slash\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:20:43.143262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143084    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-hostroot\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.143262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143138    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnw7z\" (UniqueName: \"kubernetes.io/projected/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kube-api-access-tnw7z\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.143262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143171    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/471cd70e-c62e-4f48-903d-1d71fa617d35-host\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:20:43.143262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143204    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.143262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143236    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-var-lib-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.143490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143269    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-cnibin\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.143490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143304    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bcz\" (UniqueName: \"kubernetes.io/projected/7f8bac4d-65fb-4079-ad54-a762105a2d93-kube-api-access-s2bcz\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.143490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143394    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-netd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.143490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143430    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-env-overrides\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.143490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143459    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-script-lib\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.143703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143492    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swnn\" (UniqueName: \"kubernetes.io/projected/4749d5cb-885e-418c-82ff-55ce8029e532-kube-api-access-7swnn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.143703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143521    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-tmp-dir\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:20:43.143703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143555    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-bin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.143703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143585    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-registration-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.143703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143616    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-kubelet\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.143703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143645    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-kubernetes\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.143703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143671    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9bh\" (UniqueName: \"kubernetes.io/projected/3fd62759-3b38-496d-b853-9d7bde9eb09d-kube-api-access-zp9bh\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.143703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143701    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-kubelet-config\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:43.144033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143842    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-k8s-cni-cncf-io\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.144033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143875    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-sys-fs\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.144033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143905    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-system-cni-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.144033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143936    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-conf\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.144033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143960    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-tmp\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.144033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.143996    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-system-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.144297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144028    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-etc-kubernetes\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.144297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144082    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-device-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.144297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144129    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/471cd70e-c62e-4f48-903d-1d71fa617d35-serviceca\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:20:43.144297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144235    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-tmp-dir\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:20:43.144529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144510    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-log-socket\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.144577 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144557    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f499c\" (UniqueName: \"kubernetes.io/projected/71d44c81-99b9-4df8-982a-98edaa068b3d-kube-api-access-f499c\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:43.144619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144589    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-var-lib-kubelet\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.144619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144613    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-tuned\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.144701 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144642    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rskls\" (UniqueName: \"kubernetes.io/projected/0b074486-2077-4e57-8223-f51a2bf0bea3-kube-api-access-rskls\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:20:43.144701 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144673    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:43.144807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144784    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/471cd70e-c62e-4f48-903d-1d71fa617d35-serviceca\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:20:43.144853 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144792    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-kubelet\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.144896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144848    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-etc-selinux\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.144938 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144899    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.144938 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144932    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmn4x\" (UniqueName: \"kubernetes.io/projected/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-kube-api-access-dmn4x\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:20:43.145024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144965    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-lib-modules\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.145024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.144996    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-multus\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.145340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.145316    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-env-overrides\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.145484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.145470    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-tuned\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.145828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.145807    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.145950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.145936    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-bin\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.146061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.146024    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-node-log\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.146153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.146141    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cnibin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.146243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.146232    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cni-binary-copy\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.146626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.146604    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b85m\" (UniqueName: \"kubernetes.io/projected/471cd70e-c62e-4f48-903d-1d71fa617d35-kube-api-access-2b85m\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:20:43.146979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.146961    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f8bac4d-65fb-4079-ad54-a762105a2d93-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.147080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.147035    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4749d5cb-885e-418c-82ff-55ce8029e532-ovnkube-script-lib\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.147139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.147120    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd62759-3b38-496d-b853-9d7bde9eb09d-tmp\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.147295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.147280    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cni-binary-copy\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.151212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.151194    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jmj\" (UniqueName: \"kubernetes.io/projected/5b391665-5177-4a9c-adfb-d8d9d2772ab9-kube-api-access-58jmj\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.153502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.153486    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnw7z\" (UniqueName: \"kubernetes.io/projected/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kube-api-access-tnw7z\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.157403 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.157388    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bcz\" (UniqueName: \"kubernetes.io/projected/7f8bac4d-65fb-4079-ad54-a762105a2d93-kube-api-access-s2bcz\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.157520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.157499    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swnn\" (UniqueName: \"kubernetes.io/projected/4749d5cb-885e-418c-82ff-55ce8029e532-kube-api-access-7swnn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.157556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.157517    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9bh\" (UniqueName: \"kubernetes.io/projected/3fd62759-3b38-496d-b853-9d7bde9eb09d-kube-api-access-zp9bh\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.158689 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.158673    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rskls\" (UniqueName: \"kubernetes.io/projected/0b074486-2077-4e57-8223-f51a2bf0bea3-kube-api-access-rskls\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:20:43.161797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.161780    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f499c\" (UniqueName: \"kubernetes.io/projected/71d44c81-99b9-4df8-982a-98edaa068b3d-kube-api-access-f499c\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:43.162843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.162826    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b85m\" (UniqueName: \"kubernetes.io/projected/471cd70e-c62e-4f48-903d-1d71fa617d35-kube-api-access-2b85m\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:20:43.162882 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.162869    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmn4x\" (UniqueName: \"kubernetes.io/projected/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-kube-api-access-dmn4x\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:20:43.162972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.162955    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gbm\" (UniqueName: \"kubernetes.io/projected/e0826e21-1b23-4f42-a0af-eb1621a858f5-kube-api-access-59gbm\") pod \"network-check-target-67sdw\" (UID: \"e0826e21-1b23-4f42-a0af-eb1621a858f5\") " pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:43.226814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.226767    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:20:43.226963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.226823    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 01:20:43.226963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.226840    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:20:43.226963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.226931    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gv7wl\""
Oct 31 01:20:43.227160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227090    5293 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:43.227231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227214    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:20:43.227278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227223    5293 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:43.227324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227235    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:20:43.227365 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227350    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 01:20:43.227410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227364    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9rv2t\""
Oct 31 01:20:43.227410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227354    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:20:43.227410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227401    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 01:20:43.227547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227447    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vdjwv\""
Oct 31 01:20:43.227547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227461    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbv59\""
Oct 31 01:20:43.227547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227468    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:20:43.227547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227514    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnh8f\""
Oct 31 01:20:43.227547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227513    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 01:20:43.227547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227533    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vjbgx\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227565    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d5hw4\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227578    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25cw\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227589    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227625    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bztmp\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227664    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qvf5d\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227710    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tsbmr\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227766    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-whdn9\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227807    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vj55m\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227818    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227891    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fk7jl\""
Oct 31 01:20:43.228058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.227943    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-44nrr\""
Oct 31 01:20:43.228512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.228120    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 01:20:43.228512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.228188    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-79nk6\""
Oct 31 01:20:43.228512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.228266    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6imjehf5hm31v\""
Oct 31 01:20:43.228512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.228286    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 01:20:43.228512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.228328    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82mbc\""
Oct 31 01:20:43.228512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.228376    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 01:20:43.229098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.229079    5293 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:20:43.232080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.232062    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d44c81-99b9-4df8-982a-98edaa068b3d-metrics-certs\") pod \"network-metrics-daemon-4drdm\" (UID: \"71d44c81-99b9-4df8-982a-98edaa068b3d\") " pod="openshift-multus/network-metrics-daemon-4drdm"
Oct 31 01:20:43.234234 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.234216    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb0eaa17-d3d8-4db5-b63c-1d74a7d77996-agent-certs\") pod \"konnectivity-agent-c9rzr\" (UID: \"fb0eaa17-d3d8-4db5-b63c-1d74a7d77996\") " pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:20:43.237237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.237224    5293 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:20:43.237283 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:20:43.237258    5293 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:43.240096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.240081    5293 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:20:43.240159 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:20:43.240145    5293 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-49.us-east-2.compute.internal"
Oct 31 01:20:43.247486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247468    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.247566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247494    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.247566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247512    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.247566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247529    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-sys\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.247678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247590    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-systemd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.247726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247709    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.247773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247725    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.247773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247736    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.247773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247753    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-systemd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.247889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247793    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-netd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.247889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247852    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.247889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247858    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-client-certs\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.248000 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247886    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.248000 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247899    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-netd\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.248000 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.247927    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-tls\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.248141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248071    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-cnibin\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.248183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248142    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-cnibin\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.248213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248176    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-bin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.248213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248185    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-client-certs\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.248213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248202    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-kubelet\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.248213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248208    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-secret-metrics-server-tls\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.248398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248218    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd5f332c-3b57-4a47-9dae-f191abaa5d52-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:43.248398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248233    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8wh\" (UniqueName: \"kubernetes.io/projected/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-api-access-7w8wh\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.248398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248250    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-bin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.248398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248250    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-kubelet\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.248398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.248398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248321    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-kubernetes\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.248398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248355    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-tmp-dir\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:43.248398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248380    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-kubelet-config\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248426    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-kubernetes\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248434    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248435    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd5f332c-3b57-4a47-9dae-f191abaa5d52-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248472    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-log-socket\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248473    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-kubelet-config\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248497    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-tmp-dir\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248529    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-var-lib-kubelet\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248515    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248542    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-log-socket\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248571    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-var-lib-kubelet\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248567    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfg8\" (UniqueName: \"kubernetes.io/projected/e435a839-2e87-46cf-a521-17e5ffa2dfc0-kube-api-access-bcfg8\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248593    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-kubelet\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248618    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-etc-selinux\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248635    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-lib-modules\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248631    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248649    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-cert\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248666    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-multus\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248696    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-etc-selinux\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248706    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-cni-multus\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248695    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-var-lib-kubelet\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.248710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248718    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248751    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-bin\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248768    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-lib-modules\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248777    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-node-log\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248796    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248806    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cnibin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248809    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-node-log\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248835    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-etc-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248826    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-cni-bin\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248860    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-conf-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248866    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-etc-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248838    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-cnibin\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248885    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-conf-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248888    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-systemd-units\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248912    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-cert\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248923    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-systemd-units\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248917    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-sys\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248950    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-sys\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248968    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-ovn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.248998    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249031    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxd8\" (UniqueName: \"kubernetes.io/projected/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-kube-api-access-psxd8\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249068    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-run-ovn\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249085    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-wtmp\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249113    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-dbus\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249137    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-host\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249195    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-host\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249202    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6-dbus\") pod \"global-pull-secret-syncer-sgnh6\" (UID: \"552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6\") " pod="kube-system/global-pull-secret-syncer-sgnh6"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249178    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysconfig\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249236    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249241    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcpn\" (UniqueName: \"kubernetes.io/projected/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-api-access-cwcpn\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249259    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysconfig\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249287    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-config-volume\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249314    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-socket-dir-parent\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249338    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2x5w\" (UniqueName: \"kubernetes.io/projected/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-kube-api-access-j2x5w\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249363    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b074486-2077-4e57-8223-f51a2bf0bea3-host-slash\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249384    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-hostroot\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249430    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-config-volume\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249445    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b074486-2077-4e57-8223-f51a2bf0bea3-host-slash\") pod \"iptables-alerter-fp4dj\" (UID: \"0b074486-2077-4e57-8223-f51a2bf0bea3\") " pod="openshift-network-operator/iptables-alerter-fp4dj"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249434    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-hostroot\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249480    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-socket-dir-parent\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249509    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/471cd70e-c62e-4f48-903d-1d71fa617d35-host\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249535    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrh8c\" (UniqueName: \"kubernetes.io/projected/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-kube-api-access-wrh8c\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249564    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-var-lib-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249587    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-registration-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249608    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/471cd70e-c62e-4f48-903d-1d71fa617d35-host\") pod \"node-ca-mg8qc\" (UID: \"471cd70e-c62e-4f48-903d-1d71fa617d35\") " pod="openshift-image-registry/node-ca-mg8qc"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249616    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-metrics-client-ca\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249635    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-var-lib-openvswitch\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249643    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-registration-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.249646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249647    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249684    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-metrics-tls\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249708    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249733    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5cgz\" (UniqueName: \"kubernetes.io/projected/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-kube-api-access-d5cgz\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249771    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-k8s-cni-cncf-io\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249797    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-sys-fs\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249821    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-system-cni-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249829    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249844    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-conf\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249822    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-metrics-client-ca\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249868    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-sys-fs\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249877    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-k8s-cni-cncf-io\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249905    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-system-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249912    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249934    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-metrics-tls\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249940    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f8bac4d-65fb-4079-ad54-a762105a2d93-system-cni-dir\") pod \"multus-additional-cni-plugins-ncbph\" (UID: \"7f8bac4d-65fb-4079-ad54-a762105a2d93\") " pod="openshift-multus/multus-additional-cni-plugins-ncbph"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.249925    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-etc-kubernetes\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250008    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-etc-kubernetes\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250025    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-device-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250073    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-system-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250117    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-device-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250130    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-conf\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250145    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250172    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-tls\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250211    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-hosts-file\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250237    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250261    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-netns\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250282    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-modprobe-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250289    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250311    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-root\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250325    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4950850b-9fc3-4c93-a3c8-b92ef5d906cc-hosts-file\") pod \"node-resolver-jp6vv\" (UID: \"4950850b-9fc3-4c93-a3c8-b92ef5d906cc\") " pod="openshift-dns/node-resolver-jp6vv"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250338    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250369    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-textfile\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250397    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e435a839-2e87-46cf-a521-17e5ffa2dfc0-audit-log\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250419    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-tls\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250422    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-data-volume\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250476    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250487    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e435a839-2e87-46cf-a521-17e5ffa2dfc0-audit-log\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250506    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250535    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-modprobe-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250630    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250642    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250705    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-data-volume\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250743    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-netns\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250750    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250750    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250786    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsk2\" (UniqueName: \"kubernetes.io/projected/dd5f332c-3b57-4a47-9dae-f191abaa5d52-kube-api-access-npsk2\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250792    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250813    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-crio-socket\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250830    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250851    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-textfile\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250896    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-metrics-server-audit-profiles\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250937    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-client-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250943    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd5f332c-3b57-4a47-9dae-f191abaa5d52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250956    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250962    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.250986    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251223    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e435a839-2e87-46cf-a521-17e5ffa2dfc0-metrics-server-audit-profiles\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251249    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e435a839-2e87-46cf-a521-17e5ffa2dfc0-client-ca-bundle\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251273    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251301    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-multus-cni-dir\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251303    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.251307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251330    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-systemd\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251354    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-run\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251376    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-multus-certs\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251398    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-socket-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251420    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-slash\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251437    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-sysctl-d\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251445    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-netns\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251491    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-run-netns\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251550    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-etc-systemd\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251564    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406-socket-dir\") pod \"aws-ebs-csi-driver-node-84f7d\" (UID: \"7e0e8c4d-8ba4-4f1b-914d-0d7b17ec7406\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84f7d"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251607    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fd62759-3b38-496d-b853-9d7bde9eb09d-run\") pod \"tuned-kl48l\" (UID: \"3fd62759-3b38-496d-b853-9d7bde9eb09d\") " pod="openshift-cluster-node-tuning-operator/tuned-kl48l"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251635    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4749d5cb-885e-418c-82ff-55ce8029e532-host-slash\") pod \"ovnkube-node-b2jz4\" (UID: \"4749d5cb-885e-418c-82ff-55ce8029e532\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:43.252886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.251663    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b391665-5177-4a9c-adfb-d8d9d2772ab9-host-run-multus-certs\") pod \"multus-gvxxk\" (UID: \"5b391665-5177-4a9c-adfb-d8d9d2772ab9\") " pod="openshift-multus/multus-gvxxk"
Oct 31 01:20:43.260691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.260671    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8wh\" (UniqueName: \"kubernetes.io/projected/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-kube-api-access-7w8wh\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.261285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.261269    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfg8\" (UniqueName: \"kubernetes.io/projected/e435a839-2e87-46cf-a521-17e5ffa2dfc0-kube-api-access-bcfg8\") pod \"metrics-server-5f555ffffc-q4tx9\" (UID: \"e435a839-2e87-46cf-a521-17e5ffa2dfc0\") " pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:43.261361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.261346    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxd8\" (UniqueName: \"kubernetes.io/projected/6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a-kube-api-access-psxd8\") pod \"ingress-canary-lxzrw\" (UID: \"6bc3aecc-79e1-42d2-ab4c-a4d7fcf8d70a\") " pod="openshift-ingress-canary/ingress-canary-lxzrw"
Oct 31 01:20:43.266342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.266319    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2x5w\" (UniqueName: \"kubernetes.io/projected/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-kube-api-access-j2x5w\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.266342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.266331    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrh8c\" (UniqueName: \"kubernetes.io/projected/a3cb3e38-466f-4fab-8281-1257ca8fc8ae-kube-api-access-wrh8c\") pod \"dns-default-jtnln\" (UID: \"a3cb3e38-466f-4fab-8281-1257ca8fc8ae\") " pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:43.269631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.266371    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsk2\" (UniqueName: \"kubernetes.io/projected/dd5f332c-3b57-4a47-9dae-f191abaa5d52-kube-api-access-npsk2\") pod \"openshift-state-metrics-7bd7489957-97pjd\" (UID: \"dd5f332c-3b57-4a47-9dae-f191abaa5d52\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-97pjd"
Oct 31 01:20:43.269631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.267202    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcpn\" (UniqueName: \"kubernetes.io/projected/d11ae2b3-41ce-4c4f-9e94-0fb9330700be-kube-api-access-cwcpn\") pod \"kube-state-metrics-5dc8d7d89b-hxnfz\" (UID: \"d11ae2b3-41ce-4c4f-9e94-0fb9330700be\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hxnfz"
Oct 31 01:20:43.269631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.267340    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5cgz\" (UniqueName: \"kubernetes.io/projected/a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b-kube-api-access-d5cgz\") pod \"monitoring-plugin-77f4f76594-mgfcp\" (UID: \"a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:43.352705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.352673    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-crio-socket\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.352825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.352719    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-sys\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.352825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.352768    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-wtmp\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.352825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.352786    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea8a7935-f8d1-4a05-ab04-08b62b0a9f99-crio-socket\") pod \"insights-runtime-extractor-vs2gb\" (UID: \"ea8a7935-f8d1-4a05-ab04-08b62b0a9f99\") " pod="openshift-insights/insights-runtime-extractor-vs2gb"
Oct 31 01:20:43.352825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.352815    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-root\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.352986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.352842    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-sys\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.352986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.352857    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-node-exporter-wtmp\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:43.352986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:43.352887    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88d6a153-63a8-4b4a-9ca4-e3f1796eb34b-root\") pod \"node-exporter-8cc5t\" (UID: \"88d6a153-63a8-4b4a-9ca4-e3f1796eb34b\") " pod="openshift-monitoring/node-exporter-8cc5t"
Oct 31 01:20:44.658498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:44.658461    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mgfcp"
Oct 31 01:20:45.100959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:45.100850    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:20:45.220501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:45.220475    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jtnln"
Oct 31 01:20:51.415694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:51.415668    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:51.727505 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:51.727420    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:20:51.949726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:51.949699    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f555ffffc-q4tx9"
Oct 31 01:20:52.255095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:52.255067    5293 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:20:57.129681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:20:57.129651    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c9rzr"
Oct 31 01:21:03.978647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:03.978612    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2jz4"
Oct 31 01:21:11.776057 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:21:11.775999    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:21:15.103208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:15.103174    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-67sdw"
Oct 31 01:21:41.776405 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:21:41.776359    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:21:59.072367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.072335    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5754dc898c-kx4q9"]
Oct 31 01:21:59.075794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.075773    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.079001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.078973    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 01:21:59.081311 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.081269    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 01:21:59.092097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.092025    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5754dc898c-kx4q9"]
Oct 31 01:21:59.162135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.162100    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-secret-telemeter-client\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.162324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.162141    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mq2\" (UniqueName: \"kubernetes.io/projected/08d155b0-ae14-4540-acf9-070068acc05a-kube-api-access-s9mq2\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.162324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.162227    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.162324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.162267    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-metrics-client-ca\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.162324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.162303    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-federate-client-tls\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.162540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.162405    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.162540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.162459    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-telemeter-client-tls\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.162540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.162528    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-serving-certs-ca-bundle\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.174936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.174907    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 01:21:59.174936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.174932    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 01:21:59.175093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.174965    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 01:21:59.175093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.175032    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d6dwr\""
Oct 31 01:21:59.175093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.175060    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 01:21:59.263480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263452    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.263573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263495    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-telemeter-client-tls\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.263635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263617    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-serving-certs-ca-bundle\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.263678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263657    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-secret-telemeter-client\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.263729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263679    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mq2\" (UniqueName: \"kubernetes.io/projected/08d155b0-ae14-4540-acf9-070068acc05a-kube-api-access-s9mq2\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.263729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263705    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.263891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263730    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-metrics-client-ca\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.263891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263764    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-federate-client-tls\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.263891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.263879    5293 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:21:59.264475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.264451    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-serving-certs-ca-bundle\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.264623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.264488    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.264811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.264789    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08d155b0-ae14-4540-acf9-070068acc05a-metrics-client-ca\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.266306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.266287    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-telemeter-client-tls\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.266618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.266595    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.266677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.266601    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-secret-telemeter-client\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.266887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.266870    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/08d155b0-ae14-4540-acf9-070068acc05a-federate-client-tls\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.275459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.275438    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mq2\" (UniqueName: \"kubernetes.io/projected/08d155b0-ae14-4540-acf9-070068acc05a-kube-api-access-s9mq2\") pod \"telemeter-client-5754dc898c-kx4q9\" (UID: \"08d155b0-ae14-4540-acf9-070068acc05a\") " pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.392020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.391935    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9"
Oct 31 01:21:59.671806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.671763    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5754dc898c-kx4q9"]
Oct 31 01:21:59.677791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:21:59.677771    5293 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:22:00.470362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:22:00.470321    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9" event={"ID":"08d155b0-ae14-4540-acf9-070068acc05a","Type":"ContainerStarted","Data":"9e20c1c052d6c83b24770e6cc699a85f09f71390717a6511b513d4da990ab1f6"}
Oct 31 01:22:01.475024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:22:01.474994    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9" event={"ID":"08d155b0-ae14-4540-acf9-070068acc05a","Type":"ContainerStarted","Data":"06b91c379754ea6623b7c421a162c93198e79c7dca2a1eca0d523003f8baf00f"}
Oct 31 01:22:03.483287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:22:03.483257    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9" event={"ID":"08d155b0-ae14-4540-acf9-070068acc05a","Type":"ContainerStarted","Data":"72b74d6044c994b8b9dc03881f887204fc80e1ebbecef7481d26816ee4ec66b0"}
Oct 31 01:22:03.483668 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:22:03.483311    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9" event={"ID":"08d155b0-ae14-4540-acf9-070068acc05a","Type":"ContainerStarted","Data":"583d9dcf8319d55b15ed88a169dd5716d53a4f1f141d16a1afb16949af7961aa"}
Oct 31 01:22:03.508598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:22:03.508552    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5754dc898c-kx4q9" podStartSLOduration=1.685836746 podStartE2EDuration="4.508534239s" totalImagesPullingTime="2.822697493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:03.507841587 +0000 UTC m=+81.456490015" watchObservedRunningTime="2026-10-31 01:22:03.508534239 +0000 UTC m=+81.457182625"
Oct 31 01:22:11.776611 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:22:11.776576    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:22:41.777386 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:22:41.777309    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:23:11.776213 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:23:11.776177    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:23:41.776512 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:23:41.776476    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:24:07.035420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.035384    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"]
Oct 31 01:24:07.038716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.038689    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.042966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.042946    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 01:24:07.042966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.042952    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 01:24:07.043151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.043028    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-skt79\""
Oct 31 01:24:07.053759 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.053728    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"]
Oct 31 01:24:07.070628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.070596    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"]
Oct 31 01:24:07.073907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.073889    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.089314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.089292    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"]
Oct 31 01:24:07.134641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.134616    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.134784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.134646    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.134784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.134742    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwdrt\" (UniqueName: \"kubernetes.io/projected/c6baba67-a166-4311-ae33-39695b5faf59-kube-api-access-fwdrt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.134862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.134802    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.134862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.134842    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.134936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.134907    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68n4\" (UniqueName: \"kubernetes.io/projected/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-kube-api-access-p68n4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.143208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.143189    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-lpj9w\""
Oct 31 01:24:07.235543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.235509    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdrt\" (UniqueName: \"kubernetes.io/projected/c6baba67-a166-4311-ae33-39695b5faf59-kube-api-access-fwdrt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.235543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.235544    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.235803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.235569    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.235803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.235589    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p68n4\" (UniqueName: \"kubernetes.io/projected/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-kube-api-access-p68n4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.235803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.235625    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.235803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.235663    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.236109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.236018    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.236109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.236072    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.236109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.236020    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.236212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.236107    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.250132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.250104    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68n4\" (UniqueName: \"kubernetes.io/projected/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-kube-api-access-p68n4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.250253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.250108    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwdrt\" (UniqueName: \"kubernetes.io/projected/c6baba67-a166-4311-ae33-39695b5faf59-kube-api-access-fwdrt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.353831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.353746    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:07.389115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.389082    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:07.631472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.631342    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"]
Oct 31 01:24:07.636406 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:24:07.636374    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc26695d_d0a9_4a00_8b7c_809928bf2fb9.slice/crio-ba2236a32102e47eb9fff2f02f1a5760d3a26240c2ed07e68440ed8ff43ecc3e WatchSource:0}: Error finding container ba2236a32102e47eb9fff2f02f1a5760d3a26240c2ed07e68440ed8ff43ecc3e: Status 404 returned error can't find the container with id ba2236a32102e47eb9fff2f02f1a5760d3a26240c2ed07e68440ed8ff43ecc3e
Oct 31 01:24:07.646516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.646495    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"]
Oct 31 01:24:07.653165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.653142    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.656984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.656967    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 01:24:07.661443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.661419    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 01:24:07.661547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.661457    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:24:07.661547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.661468    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vfdlj\""
Oct 31 01:24:07.661547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.661502    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 01:24:07.662337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.662322    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 01:24:07.664155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.664136    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"]
Oct 31 01:24:07.667107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.667088    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 01:24:07.670635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.670615    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"]
Oct 31 01:24:07.675416 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:24:07.675395    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6baba67_a166_4311_ae33_39695b5faf59.slice/crio-f0655b90f7427d63901e9c884486a701b292d1a4c99a3d06a3010e91d1ba4f19 WatchSource:0}: Error finding container f0655b90f7427d63901e9c884486a701b292d1a4c99a3d06a3010e91d1ba4f19: Status 404 returned error can't find the container with id f0655b90f7427d63901e9c884486a701b292d1a4c99a3d06a3010e91d1ba4f19
Oct 31 01:24:07.740083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.740058    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.740177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.740088    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57x9\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-kube-api-access-v57x9\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.740230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.740178    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.740230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.740204    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.740230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.740224    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.740333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.740241    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.740372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.740346    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d4a275-3883-4309-8057-b222323ab0d2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.841361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.841333    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d4a275-3883-4309-8057-b222323ab0d2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.841489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.841378    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.841489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.841398    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57x9\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-kube-api-access-v57x9\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.841489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.841422    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.841604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.841544    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.841604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.841595    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.841711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.841627    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.842233 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.842204    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.843812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.843780    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d4a275-3883-4309-8057-b222323ab0d2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.843938 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.843923    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.843990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.843975    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.844028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.844008    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.853838 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.853813    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.853911 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.853813    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57x9\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-kube-api-access-v57x9\") pod \"istiod-openshift-gateway-7876bfcb74-vbxl5\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:07.878415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.878390    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c" event={"ID":"fc26695d-d0a9-4a00-8b7c-809928bf2fb9","Type":"ContainerStarted","Data":"ba2236a32102e47eb9fff2f02f1a5760d3a26240c2ed07e68440ed8ff43ecc3e"}
Oct 31 01:24:07.879392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.879373    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd" event={"ID":"c6baba67-a166-4311-ae33-39695b5faf59","Type":"ContainerStarted","Data":"f0655b90f7427d63901e9c884486a701b292d1a4c99a3d06a3010e91d1ba4f19"}
Oct 31 01:24:07.964470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:07.964443    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:08.241459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:08.241381    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"]
Oct 31 01:24:08.248797 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:24:08.248771    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d4a275_3883_4309_8057_b222323ab0d2.slice/crio-52b2eed9bf5ef29b81b7c21e74ea2be15740af826c1c984fa4f71a1fc0408fa5 WatchSource:0}: Error finding container 52b2eed9bf5ef29b81b7c21e74ea2be15740af826c1c984fa4f71a1fc0408fa5: Status 404 returned error can't find the container with id 52b2eed9bf5ef29b81b7c21e74ea2be15740af826c1c984fa4f71a1fc0408fa5
Oct 31 01:24:08.885186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:08.885152    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5" event={"ID":"d6d4a275-3883-4309-8057-b222323ab0d2","Type":"ContainerStarted","Data":"52b2eed9bf5ef29b81b7c21e74ea2be15740af826c1c984fa4f71a1fc0408fa5"}
Oct 31 01:24:11.783515 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:24:11.783479    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:24:11.829337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:11.829306    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:24:11.829463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:11.829386    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:24:12.906411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:12.906376    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5" event={"ID":"d6d4a275-3883-4309-8057-b222323ab0d2","Type":"ContainerStarted","Data":"acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6"}
Oct 31 01:24:12.906947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:12.906606    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:12.908395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:12.908365    5293 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-vbxl5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:24:12.908516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:12.908424    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5" podUID="d6d4a275-3883-4309-8057-b222323ab0d2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:24:12.930572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:12.930053    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5" podStartSLOduration=2.352865445 podStartE2EDuration="5.930018777s" totalImagesPullingTime="3.577153332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:12.928009814 +0000 UTC m=+210.876658220" watchObservedRunningTime="2026-10-31 01:24:12.930018777 +0000 UTC m=+210.878667172"
Oct 31 01:24:13.911720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:13.911525    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:24:14.915669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:14.915635    5293 generic.go:356] "Generic (PLEG): container finished" podID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerID="aecdcef60ab59b0a7634eb047541366870c7898ff33a0cf3a865addbd0851052" exitCode=0
Oct 31 01:24:14.916121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:14.915694    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c" event={"ID":"fc26695d-d0a9-4a00-8b7c-809928bf2fb9","Type":"ContainerDied","Data":"aecdcef60ab59b0a7634eb047541366870c7898ff33a0cf3a865addbd0851052"}
Oct 31 01:24:16.924765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:16.924733    5293 generic.go:356] "Generic (PLEG): container finished" podID="c6baba67-a166-4311-ae33-39695b5faf59" containerID="a4bcde0612e1fb84bd8e67272114329fdceb1cc81caa718f688cb3fd704e685b" exitCode=0
Oct 31 01:24:16.925242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:16.924823    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd" event={"ID":"c6baba67-a166-4311-ae33-39695b5faf59","Type":"ContainerDied","Data":"a4bcde0612e1fb84bd8e67272114329fdceb1cc81caa718f688cb3fd704e685b"}
Oct 31 01:24:17.930758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:17.930726    5293 generic.go:356] "Generic (PLEG): container finished" podID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerID="8eb77a86198cff7d56477887c9090758d50e4fcb7010d87c834cd658f9ca15fc" exitCode=0
Oct 31 01:24:17.931139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:17.930818    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c" event={"ID":"fc26695d-d0a9-4a00-8b7c-809928bf2fb9","Type":"ContainerDied","Data":"8eb77a86198cff7d56477887c9090758d50e4fcb7010d87c834cd658f9ca15fc"}
Oct 31 01:24:19.941712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:19.941675    5293 generic.go:356] "Generic (PLEG): container finished" podID="c6baba67-a166-4311-ae33-39695b5faf59" containerID="ed7d1b5abb9cef9cd9183ceea938eee42b629d060d2bb724faddcb8c18c64656" exitCode=0
Oct 31 01:24:19.942156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:19.941763    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd" event={"ID":"c6baba67-a166-4311-ae33-39695b5faf59","Type":"ContainerDied","Data":"ed7d1b5abb9cef9cd9183ceea938eee42b629d060d2bb724faddcb8c18c64656"}
Oct 31 01:24:25.967247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:25.967213    5293 generic.go:356] "Generic (PLEG): container finished" podID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerID="f8f5558077bbb20a19be60132c95c23645e627ddc51e26ca8faf48dbd4b0dfe2" exitCode=0
Oct 31 01:24:25.967692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:25.967306    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c" event={"ID":"fc26695d-d0a9-4a00-8b7c-809928bf2fb9","Type":"ContainerDied","Data":"f8f5558077bbb20a19be60132c95c23645e627ddc51e26ca8faf48dbd4b0dfe2"}
Oct 31 01:24:25.969229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:25.969211    5293 generic.go:356] "Generic (PLEG): container finished" podID="c6baba67-a166-4311-ae33-39695b5faf59" containerID="e91c98bdfaf2d5c341d758e730a481861f94a1fe181ebfdd62940764077dc14c" exitCode=0
Oct 31 01:24:25.969330 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:25.969242    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd" event={"ID":"c6baba67-a166-4311-ae33-39695b5faf59","Type":"ContainerDied","Data":"e91c98bdfaf2d5c341d758e730a481861f94a1fe181ebfdd62940764077dc14c"}
Oct 31 01:24:27.096709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.096680    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:27.122080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.122059    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:27.217967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.217940    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-util\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-util\") pod \"c6baba67-a166-4311-ae33-39695b5faf59\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") "
Oct 31 01:24:27.218176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.218024    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-bundle\") pod \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") "
Oct 31 01:24:27.218176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.218082    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-kube-api-access-p68n4\" (UniqueName: \"kubernetes.io/projected/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-kube-api-access-p68n4\") pod \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") "
Oct 31 01:24:27.218176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.218113    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-util\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-util\") pod \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\" (UID: \"fc26695d-d0a9-4a00-8b7c-809928bf2fb9\") "
Oct 31 01:24:27.218176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.218150    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6baba67-a166-4311-ae33-39695b5faf59-kube-api-access-fwdrt\" (UniqueName: \"kubernetes.io/projected/c6baba67-a166-4311-ae33-39695b5faf59-kube-api-access-fwdrt\") pod \"c6baba67-a166-4311-ae33-39695b5faf59\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") "
Oct 31 01:24:27.218376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.218198    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-bundle\") pod \"c6baba67-a166-4311-ae33-39695b5faf59\" (UID: \"c6baba67-a166-4311-ae33-39695b5faf59\") "
Oct 31 01:24:27.218964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.218544    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-bundle" pod "fc26695d-d0a9-4a00-8b7c-809928bf2fb9" (UID: "fc26695d-d0a9-4a00-8b7c-809928bf2fb9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:27.220592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.219395    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-bundle" pod "c6baba67-a166-4311-ae33-39695b5faf59" (UID: "c6baba67-a166-4311-ae33-39695b5faf59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:27.221332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.221303    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-kube-api-access-p68n4" pod "fc26695d-d0a9-4a00-8b7c-809928bf2fb9" (UID: "fc26695d-d0a9-4a00-8b7c-809928bf2fb9"). InnerVolumeSpecName "kube-api-access-p68n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:27.224568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.224302    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-util" pod "c6baba67-a166-4311-ae33-39695b5faf59" (UID: "c6baba67-a166-4311-ae33-39695b5faf59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:27.224700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.224618    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6baba67-a166-4311-ae33-39695b5faf59-kube-api-access-fwdrt" pod "c6baba67-a166-4311-ae33-39695b5faf59" (UID: "c6baba67-a166-4311-ae33-39695b5faf59"). InnerVolumeSpecName "kube-api-access-fwdrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:27.228799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.228776    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-util" pod "fc26695d-d0a9-4a00-8b7c-809928bf2fb9" (UID: "fc26695d-d0a9-4a00-8b7c-809928bf2fb9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:27.319195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.319167    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p68n4\" (UniqueName: \"kubernetes.io/projected/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-kube-api-access-p68n4\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:27.319195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.319191    5293 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-util\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:27.319357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.319201    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwdrt\" (UniqueName: \"kubernetes.io/projected/c6baba67-a166-4311-ae33-39695b5faf59-kube-api-access-fwdrt\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:27.319357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.319210    5293 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-bundle\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:27.319357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.319218    5293 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6baba67-a166-4311-ae33-39695b5faf59-util\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:27.319357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.319226    5293 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc26695d-d0a9-4a00-8b7c-809928bf2fb9-bundle\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:27.978636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.978601    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd"
Oct 31 01:24:27.978636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.978612    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvgkrd" event={"ID":"c6baba67-a166-4311-ae33-39695b5faf59","Type":"ContainerDied","Data":"f0655b90f7427d63901e9c884486a701b292d1a4c99a3d06a3010e91d1ba4f19"}
Oct 31 01:24:27.978888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.978697    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0655b90f7427d63901e9c884486a701b292d1a4c99a3d06a3010e91d1ba4f19"
Oct 31 01:24:27.980479 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.980458    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c" event={"ID":"fc26695d-d0a9-4a00-8b7c-809928bf2fb9","Type":"ContainerDied","Data":"ba2236a32102e47eb9fff2f02f1a5760d3a26240c2ed07e68440ed8ff43ecc3e"}
Oct 31 01:24:27.980589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.980492    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba2236a32102e47eb9fff2f02f1a5760d3a26240c2ed07e68440ed8ff43ecc3e"
Oct 31 01:24:27.980589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:27.980492    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444x99c"
Oct 31 01:24:41.775994 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:24:41.775952    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:24:42.349449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349418    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"]
Oct 31 01:24:42.349818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349801    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerName="extract"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349820    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerName="extract"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349841    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerName="pull"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349847    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerName="pull"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349863    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c6baba67-a166-4311-ae33-39695b5faf59" containerName="util"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349871    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6baba67-a166-4311-ae33-39695b5faf59" containerName="util"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349882    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c6baba67-a166-4311-ae33-39695b5faf59" containerName="pull"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349890    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6baba67-a166-4311-ae33-39695b5faf59" containerName="pull"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349918    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c6baba67-a166-4311-ae33-39695b5faf59" containerName="extract"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349926    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6baba67-a166-4311-ae33-39695b5faf59" containerName="extract"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349944    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerName="util"
Oct 31 01:24:42.349948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.349951    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerName="util"
Oct 31 01:24:42.350595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.350068    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc26695d-d0a9-4a00-8b7c-809928bf2fb9" containerName="extract"
Oct 31 01:24:42.350595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.350082    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6baba67-a166-4311-ae33-39695b5faf59" containerName="extract"
Oct 31 01:24:42.352857 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.352838    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.364469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.364451    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 01:24:42.377894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.377876    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-25fpn\""
Oct 31 01:24:42.377997 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.377905    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 01:24:42.377997 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.377926    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 01:24:42.377997 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.377984    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 01:24:42.378221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.378206    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 01:24:42.444327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.444298    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/427f397d-3c8c-4aff-8659-51ed3722c6ca-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.444424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.444331    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.444424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.444368    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.444550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.444460    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/427f397d-3c8c-4aff-8659-51ed3722c6ca-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.444550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.444488    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krpx7\" (UniqueName: \"kubernetes.io/projected/427f397d-3c8c-4aff-8659-51ed3722c6ca-kube-api-access-krpx7\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.444616 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.444574    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.492866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.492816    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"]
Oct 31 01:24:42.545291 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.545263    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.545431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.545303    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/427f397d-3c8c-4aff-8659-51ed3722c6ca-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.545431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.545320    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.545431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.545375    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.545431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.545403    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/427f397d-3c8c-4aff-8659-51ed3722c6ca-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.545628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.545432    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krpx7\" (UniqueName: \"kubernetes.io/projected/427f397d-3c8c-4aff-8659-51ed3722c6ca-kube-api-access-krpx7\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.545761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.545737    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/427f397d-3c8c-4aff-8659-51ed3722c6ca-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.547972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.547943    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/427f397d-3c8c-4aff-8659-51ed3722c6ca-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.547972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.547961    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.548145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.548116    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.548145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.548125    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/427f397d-3c8c-4aff-8659-51ed3722c6ca-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.564882 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.564857    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krpx7\" (UniqueName: \"kubernetes.io/projected/427f397d-3c8c-4aff-8659-51ed3722c6ca-kube-api-access-krpx7\") pod \"opendatahub-operator-controller-manager-69496c6ffb-pm9wz\" (UID: \"427f397d-3c8c-4aff-8659-51ed3722c6ca\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:42.665536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:42.665505    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:43.010326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:43.010248    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"]
Oct 31 01:24:43.013271 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:24:43.013244    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427f397d_3c8c_4aff_8659_51ed3722c6ca.slice/crio-2474daf1e854b2464f9249543712165d93844a505634788760e0dee64a6fa896 WatchSource:0}: Error finding container 2474daf1e854b2464f9249543712165d93844a505634788760e0dee64a6fa896: Status 404 returned error can't find the container with id 2474daf1e854b2464f9249543712165d93844a505634788760e0dee64a6fa896
Oct 31 01:24:43.032847 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:43.032816    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz" event={"ID":"427f397d-3c8c-4aff-8659-51ed3722c6ca","Type":"ContainerStarted","Data":"2474daf1e854b2464f9249543712165d93844a505634788760e0dee64a6fa896"}
Oct 31 01:24:48.053451 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:48.053368    5293 generic.go:356] "Generic (PLEG): container finished" podID="427f397d-3c8c-4aff-8659-51ed3722c6ca" containerID="7d8591485bef9fa5eac84dc82fbe615cb4902deb03db6ca4e0c9cbc7cde5bd2e" exitCode=0
Oct 31 01:24:48.053451 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:48.053425    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz" event={"ID":"427f397d-3c8c-4aff-8659-51ed3722c6ca","Type":"ContainerDied","Data":"7d8591485bef9fa5eac84dc82fbe615cb4902deb03db6ca4e0c9cbc7cde5bd2e"}
Oct 31 01:24:49.059323 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:49.059284    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz" event={"ID":"427f397d-3c8c-4aff-8659-51ed3722c6ca","Type":"ContainerStarted","Data":"85d7a4783af4a95933ea87a8bfc0f81f21e26da5fe53fa15d13fad52056b1b47"}
Oct 31 01:24:49.059708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:49.059390    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:24:49.119237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:24:49.119188    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz" podStartSLOduration=1.9838773349999999 podStartE2EDuration="7.119173823s" totalImagesPullingTime="4.726905517s" totalInitContainerRuntime="408.390971ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:49.116232295 +0000 UTC m=+247.064880688" watchObservedRunningTime="2026-10-31 01:24:49.119173823 +0000 UTC m=+247.067822215"
Oct 31 01:25:00.067924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:00.067892    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-pm9wz"
Oct 31 01:25:07.787475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.787439    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"]
Oct 31 01:25:07.791564 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.791545    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:07.796903 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.796878    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 01:25:07.797347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.797328    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 01:25:07.797545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.797531    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sxbgr\""
Oct 31 01:25:07.806722 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.806685    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"]
Oct 31 01:25:07.838569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.838536    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rr6\" (UniqueName: \"kubernetes.io/projected/987f6046-5693-4632-b4f5-3197514c733f-kube-api-access-88rr6\") pod \"kuadrant-operator-controller-manager-678d87766f-7l4hn\" (UID: \"987f6046-5693-4632-b4f5-3197514c733f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:07.838684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.838579    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987f6046-5693-4632-b4f5-3197514c733f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7l4hn\" (UID: \"987f6046-5693-4632-b4f5-3197514c733f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:07.939618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.939584    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88rr6\" (UniqueName: \"kubernetes.io/projected/987f6046-5693-4632-b4f5-3197514c733f-kube-api-access-88rr6\") pod \"kuadrant-operator-controller-manager-678d87766f-7l4hn\" (UID: \"987f6046-5693-4632-b4f5-3197514c733f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:07.939618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.939625    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987f6046-5693-4632-b4f5-3197514c733f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7l4hn\" (UID: \"987f6046-5693-4632-b4f5-3197514c733f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:07.939950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.939933    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987f6046-5693-4632-b4f5-3197514c733f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7l4hn\" (UID: \"987f6046-5693-4632-b4f5-3197514c733f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:07.951838 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:07.951811    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rr6\" (UniqueName: \"kubernetes.io/projected/987f6046-5693-4632-b4f5-3197514c733f-kube-api-access-88rr6\") pod \"kuadrant-operator-controller-manager-678d87766f-7l4hn\" (UID: \"987f6046-5693-4632-b4f5-3197514c733f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:08.103878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:08.103792    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:08.383133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:08.383069    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"]
Oct 31 01:25:08.388763 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:25:08.388735    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987f6046_5693_4632_b4f5_3197514c733f.slice/crio-1716e687d3245353200c286f90ba7b6e2e23bdb18a6f36255d05f02e6b06614b WatchSource:0}: Error finding container 1716e687d3245353200c286f90ba7b6e2e23bdb18a6f36255d05f02e6b06614b: Status 404 returned error can't find the container with id 1716e687d3245353200c286f90ba7b6e2e23bdb18a6f36255d05f02e6b06614b
Oct 31 01:25:09.131242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:09.131211    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn" event={"ID":"987f6046-5693-4632-b4f5-3197514c733f","Type":"ContainerStarted","Data":"1716e687d3245353200c286f90ba7b6e2e23bdb18a6f36255d05f02e6b06614b"}
Oct 31 01:25:11.782539 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:25:11.782496    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:25:13.495391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.495254    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"]
Oct 31 01:25:13.499934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.499902    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.505352 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.505327    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 01:25:13.506560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.505562    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 01:25:13.506560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.505627    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jt94z\""
Oct 31 01:25:13.511239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.511209    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"]
Oct 31 01:25:13.590811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.590782    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6956b96-691a-445b-89e2-2fe5b289674e-tmp\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.590992 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.590865    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7nv\" (UniqueName: \"kubernetes.io/projected/d6956b96-691a-445b-89e2-2fe5b289674e-kube-api-access-zh7nv\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.590992 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.590962    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6956b96-691a-445b-89e2-2fe5b289674e-tls-certs\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.692426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.692380    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6956b96-691a-445b-89e2-2fe5b289674e-tmp\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.692599 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.692472    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7nv\" (UniqueName: \"kubernetes.io/projected/d6956b96-691a-445b-89e2-2fe5b289674e-kube-api-access-zh7nv\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.692599 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.692508    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6956b96-691a-445b-89e2-2fe5b289674e-tls-certs\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.695645 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.695619    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6956b96-691a-445b-89e2-2fe5b289674e-tls-certs\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.696027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.695995    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6956b96-691a-445b-89e2-2fe5b289674e-tmp\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.723210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.723146    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7nv\" (UniqueName: \"kubernetes.io/projected/d6956b96-691a-445b-89e2-2fe5b289674e-kube-api-access-zh7nv\") pod \"kube-auth-proxy-6b44bd95d7-tdrn5\" (UID: \"d6956b96-691a-445b-89e2-2fe5b289674e\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:13.817234 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:13.817152    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"
Oct 31 01:25:14.124217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:14.124188    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5"]
Oct 31 01:25:14.128240 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:25:14.128202    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6956b96_691a_445b_89e2_2fe5b289674e.slice/crio-2b8c59201568e4b64907be02e0b717cb8f1efb616b1390c78e10be87d1cc2a4d WatchSource:0}: Error finding container 2b8c59201568e4b64907be02e0b717cb8f1efb616b1390c78e10be87d1cc2a4d: Status 404 returned error can't find the container with id 2b8c59201568e4b64907be02e0b717cb8f1efb616b1390c78e10be87d1cc2a4d
Oct 31 01:25:14.155865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:14.155834    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn" event={"ID":"987f6046-5693-4632-b4f5-3197514c733f","Type":"ContainerStarted","Data":"543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b"}
Oct 31 01:25:14.156005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:14.155952    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:14.157167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:14.157141    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5" event={"ID":"d6956b96-691a-445b-89e2-2fe5b289674e","Type":"ContainerStarted","Data":"2b8c59201568e4b64907be02e0b717cb8f1efb616b1390c78e10be87d1cc2a4d"}
Oct 31 01:25:14.178136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:14.177556    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn" podStartSLOduration=1.509898261 podStartE2EDuration="7.17753857s" totalImagesPullingTime="5.667640309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:14.176470888 +0000 UTC m=+272.125119280" watchObservedRunningTime="2026-10-31 01:25:14.17753857 +0000 UTC m=+272.126186961"
Oct 31 01:25:17.171628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:17.171595    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5" event={"ID":"d6956b96-691a-445b-89e2-2fe5b289674e","Type":"ContainerStarted","Data":"bcb482d068a9f52a876216637bb529696fba47c8408cf7556c4dd76cd4bf0e29"}
Oct 31 01:25:17.199282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:17.199214    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-tdrn5" podStartSLOduration=1.47051397 podStartE2EDuration="4.199200047s" totalImagesPullingTime="2.728686077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:17.198173415 +0000 UTC m=+275.146821808" watchObservedRunningTime="2026-10-31 01:25:17.199200047 +0000 UTC m=+275.147848439"
Oct 31 01:25:25.163670 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:25:25.163601    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:25:41.776385 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:25:41.776349    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:26:10.577468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.577440    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s9rxb"]
Oct 31 01:26:10.580607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.580590    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:10.597708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.597671    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 01:26:10.597900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.597847    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 01:26:10.599751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.599731    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-4bjhg"]
Oct 31 01:26:10.603731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.603709    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s9rxb"]
Oct 31 01:26:10.603845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.603826    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.615257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.615235    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-4bjhg"]
Oct 31 01:26:10.624550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.624531    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-sqcmq\""
Oct 31 01:26:10.624980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.624926    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 01:26:10.624980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.624943    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ldpgl\""
Oct 31 01:26:10.624980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.624949    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 31 01:26:10.625193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.625138    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 01:26:10.677925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.677888    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e75104bf-ae55-488f-bab5-595defac1e17-tls-certs\") pod \"model-serving-api-75bd89fdfc-s9rxb\" (UID: \"e75104bf-ae55-488f-bab5-595defac1e17\") " pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:10.678076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.677963    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3ae4d1-4d61-457c-a552-b4001c770117-metrics-certs\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.678128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.678078    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25bm\" (UniqueName: \"kubernetes.io/projected/e75104bf-ae55-488f-bab5-595defac1e17-kube-api-access-m25bm\") pod \"model-serving-api-75bd89fdfc-s9rxb\" (UID: \"e75104bf-ae55-488f-bab5-595defac1e17\") " pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:10.678128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.678106    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba3ae4d1-4d61-457c-a552-b4001c770117-cert\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.678128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.678124    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gh2\" (UniqueName: \"kubernetes.io/projected/ba3ae4d1-4d61-457c-a552-b4001c770117-kube-api-access-n4gh2\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.779080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.779035    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba3ae4d1-4d61-457c-a552-b4001c770117-cert\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.779244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.779093    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gh2\" (UniqueName: \"kubernetes.io/projected/ba3ae4d1-4d61-457c-a552-b4001c770117-kube-api-access-n4gh2\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.779244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.779136    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e75104bf-ae55-488f-bab5-595defac1e17-tls-certs\") pod \"model-serving-api-75bd89fdfc-s9rxb\" (UID: \"e75104bf-ae55-488f-bab5-595defac1e17\") " pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:10.779244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.779206    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3ae4d1-4d61-457c-a552-b4001c770117-metrics-certs\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.779411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.779265    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25bm\" (UniqueName: \"kubernetes.io/projected/e75104bf-ae55-488f-bab5-595defac1e17-kube-api-access-m25bm\") pod \"model-serving-api-75bd89fdfc-s9rxb\" (UID: \"e75104bf-ae55-488f-bab5-595defac1e17\") " pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:10.781854 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.781829    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e75104bf-ae55-488f-bab5-595defac1e17-tls-certs\") pod \"model-serving-api-75bd89fdfc-s9rxb\" (UID: \"e75104bf-ae55-488f-bab5-595defac1e17\") " pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:10.781963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.781834    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3ae4d1-4d61-457c-a552-b4001c770117-metrics-certs\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.781963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.781936    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba3ae4d1-4d61-457c-a552-b4001c770117-cert\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.793728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.793700    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gh2\" (UniqueName: \"kubernetes.io/projected/ba3ae4d1-4d61-457c-a552-b4001c770117-kube-api-access-n4gh2\") pod \"odh-model-controller-769875db56-4bjhg\" (UID: \"ba3ae4d1-4d61-457c-a552-b4001c770117\") " pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:10.801552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.801531    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25bm\" (UniqueName: \"kubernetes.io/projected/e75104bf-ae55-488f-bab5-595defac1e17-kube-api-access-m25bm\") pod \"model-serving-api-75bd89fdfc-s9rxb\" (UID: \"e75104bf-ae55-488f-bab5-595defac1e17\") " pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:10.893376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.893306    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:10.917013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:10.916988    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:11.168945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.168915    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s9rxb"]
Oct 31 01:26:11.172209 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:26:11.172181    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75104bf_ae55_488f_bab5_595defac1e17.slice/crio-25e6f6235e3b3a75aa9e23b12363e1ccc2fdec63dd981ae0537279b52d9b7171 WatchSource:0}: Error finding container 25e6f6235e3b3a75aa9e23b12363e1ccc2fdec63dd981ae0537279b52d9b7171: Status 404 returned error can't find the container with id 25e6f6235e3b3a75aa9e23b12363e1ccc2fdec63dd981ae0537279b52d9b7171
Oct 31 01:26:11.201471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.201442    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-4bjhg"]
Oct 31 01:26:11.205782 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:26:11.205750    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3ae4d1_4d61_457c_a552_b4001c770117.slice/crio-1fac277bd10a018ac604ba34131adc8d191d74967c0c4d9c8e0e04c3033b1d6c WatchSource:0}: Error finding container 1fac277bd10a018ac604ba34131adc8d191d74967c0c4d9c8e0e04c3033b1d6c: Status 404 returned error can't find the container with id 1fac277bd10a018ac604ba34131adc8d191d74967c0c4d9c8e0e04c3033b1d6c
Oct 31 01:26:11.340175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.340146    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"]
Oct 31 01:26:11.344511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.344491    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.362842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.362817    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"]
Oct 31 01:26:11.378962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.378924    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4bjhg" event={"ID":"ba3ae4d1-4d61-457c-a552-b4001c770117","Type":"ContainerStarted","Data":"1fac277bd10a018ac604ba34131adc8d191d74967c0c4d9c8e0e04c3033b1d6c"}
Oct 31 01:26:11.380091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.380064    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb" event={"ID":"e75104bf-ae55-488f-bab5-595defac1e17","Type":"ContainerStarted","Data":"25e6f6235e3b3a75aa9e23b12363e1ccc2fdec63dd981ae0537279b52d9b7171"}
Oct 31 01:26:11.487303 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.487221    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.487303 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.487257    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6a6c52-4b49-4647-b86f-b2a91537032c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.487303 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.487282    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.487529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.487355    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.487529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.487418    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92lj\" (UniqueName: \"kubernetes.io/projected/1b6a6c52-4b49-4647-b86f-b2a91537032c-kube-api-access-m92lj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.487529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.487456    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.487529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.487474    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.587840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.587804    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.587840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.587839    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6a6c52-4b49-4647-b86f-b2a91537032c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.588351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.587861    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.588351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.587885    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.588351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.587928    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92lj\" (UniqueName: \"kubernetes.io/projected/1b6a6c52-4b49-4647-b86f-b2a91537032c-kube-api-access-m92lj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.588351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.587966    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.588351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.588104    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.588851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.588824    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.590385 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.590367    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.590450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.590397    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.590824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.590793    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6a6c52-4b49-4647-b86f-b2a91537032c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.590902 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.590871    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.599837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.599818    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6a6c52-4b49-4647-b86f-b2a91537032c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.600007 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.599990    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92lj\" (UniqueName: \"kubernetes.io/projected/1b6a6c52-4b49-4647-b86f-b2a91537032c-kube-api-access-m92lj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5p96n\" (UID: \"1b6a6c52-4b49-4647-b86f-b2a91537032c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.657078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.657055    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:11.785491 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:26:11.785406    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:26:11.984397 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.984223    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"]
Oct 31 01:26:11.994561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.991515    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:26:11.994561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:11.991602    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:26:12.389157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:12.389070    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n" event={"ID":"1b6a6c52-4b49-4647-b86f-b2a91537032c","Type":"ContainerStarted","Data":"673bf9c16553ba9b652b24016aa672d7d2fb3ad919370492f30c1b1a670b73b9"}
Oct 31 01:26:12.389157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:12.389139    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n" event={"ID":"1b6a6c52-4b49-4647-b86f-b2a91537032c","Type":"ContainerStarted","Data":"70fe0389f977dd5274e0f80fb53bbbeeda1dfd317b2efa09d67c6cbff2174271"}
Oct 31 01:26:12.389881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:12.389858    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:13.395523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.395482    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n"
Oct 31 01:26:13.424204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.424151    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5p96n" podStartSLOduration=2.424131636 podStartE2EDuration="2.424131636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:12.428914187 +0000 UTC m=+330.377562580" watchObservedRunningTime="2026-10-31 01:26:13.424131636 +0000 UTC m=+331.372780027"
Oct 31 01:26:13.456502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.456470    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"]
Oct 31 01:26:13.456876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.456847    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5" podUID="d6d4a275-3883-4309-8057-b222323ab0d2" containerName="discovery" containerID="cri-o://acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6" gracePeriod=30
Oct 31 01:26:13.753794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.753767    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:26:13.915100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.914993    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-kube-api-access-v57x9\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-kube-api-access-v57x9\") pod \"d6d4a275-3883-4309-8057-b222323ab0d2\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") "
Oct 31 01:26:13.915248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.915103    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-ca-configmap\") pod \"d6d4a275-3883-4309-8057-b222323ab0d2\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") "
Oct 31 01:26:13.915248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.915160    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-istio-token\") pod \"d6d4a275-3883-4309-8057-b222323ab0d2\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") "
Oct 31 01:26:13.915248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.915183    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-dns-cert\") pod \"d6d4a275-3883-4309-8057-b222323ab0d2\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") "
Oct 31 01:26:13.915248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.915218    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-cacerts\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-cacerts\") pod \"d6d4a275-3883-4309-8057-b222323ab0d2\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") "
Oct 31 01:26:13.915248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.915249    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-kubeconfig\") pod \"d6d4a275-3883-4309-8057-b222323ab0d2\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") "
Oct 31 01:26:13.915508 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.915293    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d4a275-3883-4309-8057-b222323ab0d2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d4a275-3883-4309-8057-b222323ab0d2-local-certs\") pod \"d6d4a275-3883-4309-8057-b222323ab0d2\" (UID: \"d6d4a275-3883-4309-8057-b222323ab0d2\") "
Oct 31 01:26:13.916201 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.916020    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-ca-configmap" pod "d6d4a275-3883-4309-8057-b222323ab0d2" (UID: "d6d4a275-3883-4309-8057-b222323ab0d2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:13.918787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.918751    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-istio-token" pod "d6d4a275-3883-4309-8057-b222323ab0d2" (UID: "d6d4a275-3883-4309-8057-b222323ab0d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:13.919277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.919189    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-kube-api-access-v57x9" pod "d6d4a275-3883-4309-8057-b222323ab0d2" (UID: "d6d4a275-3883-4309-8057-b222323ab0d2"). InnerVolumeSpecName "kube-api-access-v57x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:13.919277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.919221    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d4a275-3883-4309-8057-b222323ab0d2-local-certs" pod "d6d4a275-3883-4309-8057-b222323ab0d2" (UID: "d6d4a275-3883-4309-8057-b222323ab0d2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:13.919277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.919232    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-cacerts" pod "d6d4a275-3883-4309-8057-b222323ab0d2" (UID: "d6d4a275-3883-4309-8057-b222323ab0d2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:13.920515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.920481    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-dns-cert" pod "d6d4a275-3883-4309-8057-b222323ab0d2" (UID: "d6d4a275-3883-4309-8057-b222323ab0d2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:13.920695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:13.920670    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-kubeconfig" pod "d6d4a275-3883-4309-8057-b222323ab0d2" (UID: "d6d4a275-3883-4309-8057-b222323ab0d2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:14.016260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.016215    5293 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-kubeconfig\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:14.016260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.016255    5293 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d4a275-3883-4309-8057-b222323ab0d2-local-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:14.016260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.016266    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57x9\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-kube-api-access-v57x9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:14.016477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.016277    5293 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-ca-configmap\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:14.016477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.016286    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d4a275-3883-4309-8057-b222323ab0d2-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:14.016477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.016294    5293 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-istio-csr-dns-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:14.016477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.016303    5293 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6d4a275-3883-4309-8057-b222323ab0d2-cacerts\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:14.400094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.400061    5293 generic.go:356] "Generic (PLEG): container finished" podID="d6d4a275-3883-4309-8057-b222323ab0d2" containerID="acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6" exitCode=0
Oct 31 01:26:14.400539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.400080    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5" event={"ID":"d6d4a275-3883-4309-8057-b222323ab0d2","Type":"ContainerDied","Data":"acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6"}
Oct 31 01:26:14.400539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.400111    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"
Oct 31 01:26:14.400539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.400159    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5" event={"ID":"d6d4a275-3883-4309-8057-b222323ab0d2","Type":"ContainerDied","Data":"52b2eed9bf5ef29b81b7c21e74ea2be15740af826c1c984fa4f71a1fc0408fa5"}
Oct 31 01:26:14.400539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.400186    5293 scope.go:122] "RemoveContainer" containerID="acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6"
Oct 31 01:26:14.403114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.403089    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb" event={"ID":"e75104bf-ae55-488f-bab5-595defac1e17","Type":"ContainerStarted","Data":"afe831fb362b1f0b99107eb3489aca50576b110f95b64d8d2c2d5abbebf407aa"}
Oct 31 01:26:14.403239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.403142    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:14.410566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.410546    5293 scope.go:122] "RemoveContainer" containerID="acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6"
Oct 31 01:26:14.410899 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:26:14.410871    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6\": container with ID starting with acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6 not found: ID does not exist" containerID="acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6"
Oct 31 01:26:14.410976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.410925    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6"} err="failed to get container status \"acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6\": rpc error: code = NotFound desc = could not find container \"acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6\": container with ID starting with acc9bc757304d661015c028a503c50aa330520ac58b20bb6a7eb4b045ebd5ac6 not found: ID does not exist"
Oct 31 01:26:14.423057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.422973    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb" podStartSLOduration=1.945333873 podStartE2EDuration="4.422957871s" totalImagesPullingTime="2.477623998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:14.421079639 +0000 UTC m=+332.369728022" watchObservedRunningTime="2026-10-31 01:26:14.422957871 +0000 UTC m=+332.371606264"
Oct 31 01:26:14.435156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.435092    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"]
Oct 31 01:26:14.438981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:14.438957    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vbxl5"]
Oct 31 01:26:15.407468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:15.407425    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4bjhg" event={"ID":"ba3ae4d1-4d61-457c-a552-b4001c770117","Type":"ContainerStarted","Data":"85e2161f08c88d814550f7104858fca8bee5879e2f0b6abf6d42b6c598bf77d4"}
Oct 31 01:26:15.407905 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:15.407503    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:15.427613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:15.427552    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-4bjhg" podStartSLOduration=2.100974532 podStartE2EDuration="5.427534814s" totalImagesPullingTime="3.326560282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:15.427033343 +0000 UTC m=+333.375681736" watchObservedRunningTime="2026-10-31 01:26:15.427534814 +0000 UTC m=+333.376183207"
Oct 31 01:26:16.171467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:16.171436    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d4a275-3883-4309-8057-b222323ab0d2" path="/var/lib/kubelet/pods/d6d4a275-3883-4309-8057-b222323ab0d2/volumes"
Oct 31 01:26:25.412495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:25.412463    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb"
Oct 31 01:26:26.416572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:26:26.416545    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-4bjhg"
Oct 31 01:26:41.776375 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:26:41.776332    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:27:11.777358 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:27:11.777318    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:27:41.777197 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:27:41.777163    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:28:11.778224 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:28:11.778179    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:28:41.776155 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:28:41.776082    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:29:11.777017 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:29:11.776981    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:29:41.776491 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:29:41.776389    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:30:11.776760 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:30:11.776718    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:30:41.776783 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:30:41.776742    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:31:11.776359 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:31:11.776309    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:31:41.777193 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:31:41.777118    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:32:11.775775 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:32:11.775735    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:32:41.776821 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:32:41.776786    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:33:11.777710 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:33:11.777676    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:33:41.777120 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:33:41.777081    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:34:11.777027 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:34:11.776991    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:34:41.777075 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:34:41.776951    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:35:11.776761 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:35:11.776728    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:35:34.487719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.487690    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7l4hn_987f6046-5693-4632-b4f5-3197514c733f/manager/0.log"
Oct 31 01:35:34.709441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.709406    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"]
Oct 31 01:35:34.709652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.709629    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn" podUID="987f6046-5693-4632-b4f5-3197514c733f" containerName="manager" containerID="cri-o://543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b" gracePeriod=10
Oct 31 01:35:34.744370 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.744301    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"]
Oct 31 01:35:34.744756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.744737    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d6d4a275-3883-4309-8057-b222323ab0d2" containerName="discovery"
Oct 31 01:35:34.744841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.744758    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d4a275-3883-4309-8057-b222323ab0d2" containerName="discovery"
Oct 31 01:35:34.744878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.744852    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d4a275-3883-4309-8057-b222323ab0d2" containerName="discovery"
Oct 31 01:35:34.747696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.747675    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:34.764539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.764515    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"]
Oct 31 01:35:34.845258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.845219    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/771a7056-0944-443b-969c-6d4283bec9bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-46qd9\" (UID: \"771a7056-0944-443b-969c-6d4283bec9bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:34.845390 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.845370    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jg87\" (UniqueName: \"kubernetes.io/projected/771a7056-0944-443b-969c-6d4283bec9bf-kube-api-access-9jg87\") pod \"kuadrant-operator-controller-manager-678d87766f-46qd9\" (UID: \"771a7056-0944-443b-969c-6d4283bec9bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:34.946482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.946461    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:35:34.946632 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.946611    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jg87\" (UniqueName: \"kubernetes.io/projected/771a7056-0944-443b-969c-6d4283bec9bf-kube-api-access-9jg87\") pod \"kuadrant-operator-controller-manager-678d87766f-46qd9\" (UID: \"771a7056-0944-443b-969c-6d4283bec9bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:34.946705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.946652    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/771a7056-0944-443b-969c-6d4283bec9bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-46qd9\" (UID: \"771a7056-0944-443b-969c-6d4283bec9bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:34.946929 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.946913    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/771a7056-0944-443b-969c-6d4283bec9bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-46qd9\" (UID: \"771a7056-0944-443b-969c-6d4283bec9bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:34.958334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:34.958308    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jg87\" (UniqueName: \"kubernetes.io/projected/771a7056-0944-443b-969c-6d4283bec9bf-kube-api-access-9jg87\") pod \"kuadrant-operator-controller-manager-678d87766f-46qd9\" (UID: \"771a7056-0944-443b-969c-6d4283bec9bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:35.047751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.047675    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987f6046-5693-4632-b4f5-3197514c733f-kube-api-access-88rr6\" (UniqueName: \"kubernetes.io/projected/987f6046-5693-4632-b4f5-3197514c733f-kube-api-access-88rr6\") pod \"987f6046-5693-4632-b4f5-3197514c733f\" (UID: \"987f6046-5693-4632-b4f5-3197514c733f\") "
Oct 31 01:35:35.047883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.047762    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987f6046-5693-4632-b4f5-3197514c733f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987f6046-5693-4632-b4f5-3197514c733f-extensions-socket-volume\") pod \"987f6046-5693-4632-b4f5-3197514c733f\" (UID: \"987f6046-5693-4632-b4f5-3197514c733f\") "
Oct 31 01:35:35.048152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.048124    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987f6046-5693-4632-b4f5-3197514c733f-extensions-socket-volume" pod "987f6046-5693-4632-b4f5-3197514c733f" (UID: "987f6046-5693-4632-b4f5-3197514c733f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:35.049938 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.049913    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987f6046-5693-4632-b4f5-3197514c733f-kube-api-access-88rr6" pod "987f6046-5693-4632-b4f5-3197514c733f" (UID: "987f6046-5693-4632-b4f5-3197514c733f"). InnerVolumeSpecName "kube-api-access-88rr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:35.108087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.108062    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:35.148591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.148564    5293 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987f6046-5693-4632-b4f5-3197514c733f-extensions-socket-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:35.148591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.148589    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88rr6\" (UniqueName: \"kubernetes.io/projected/987f6046-5693-4632-b4f5-3197514c733f-kube-api-access-88rr6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:35.393346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.393266    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"]
Oct 31 01:35:35.397271 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:35:35.397236    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771a7056_0944_443b_969c_6d4283bec9bf.slice/crio-487b09920f4ba57bf0e0c601c699fbf3908e087d4e490f7bbf62e6dad9eacc7e WatchSource:0}: Error finding container 487b09920f4ba57bf0e0c601c699fbf3908e087d4e490f7bbf62e6dad9eacc7e: Status 404 returned error can't find the container with id 487b09920f4ba57bf0e0c601c699fbf3908e087d4e490f7bbf62e6dad9eacc7e
Oct 31 01:35:35.574706 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.574672    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9" event={"ID":"771a7056-0944-443b-969c-6d4283bec9bf","Type":"ContainerStarted","Data":"16dd953725d0bfd7d76504216883e766f7909013401acc102998001dff21292d"}
Oct 31 01:35:35.575187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.574728    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9" event={"ID":"771a7056-0944-443b-969c-6d4283bec9bf","Type":"ContainerStarted","Data":"487b09920f4ba57bf0e0c601c699fbf3908e087d4e490f7bbf62e6dad9eacc7e"}
Oct 31 01:35:35.575187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.574796    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:35.576130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.576106    5293 generic.go:356] "Generic (PLEG): container finished" podID="987f6046-5693-4632-b4f5-3197514c733f" containerID="543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b" exitCode=0
Oct 31 01:35:35.576241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.576145    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"
Oct 31 01:35:35.576241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.576185    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn" event={"ID":"987f6046-5693-4632-b4f5-3197514c733f","Type":"ContainerDied","Data":"543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b"}
Oct 31 01:35:35.576241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.576222    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn" event={"ID":"987f6046-5693-4632-b4f5-3197514c733f","Type":"ContainerDied","Data":"1716e687d3245353200c286f90ba7b6e2e23bdb18a6f36255d05f02e6b06614b"}
Oct 31 01:35:35.576332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.576248    5293 scope.go:122] "RemoveContainer" containerID="543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b"
Oct 31 01:35:35.585468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.585455    5293 scope.go:122] "RemoveContainer" containerID="543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b"
Oct 31 01:35:35.585699 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:35:35.585679    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b\": container with ID starting with 543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b not found: ID does not exist" containerID="543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b"
Oct 31 01:35:35.585740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.585708    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b"} err="failed to get container status \"543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b\": rpc error: code = NotFound desc = could not find container \"543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b\": container with ID starting with 543b7462cdd10ba20d6fa87046d8b49b3eb9d475a99f9690968f65fd8d14a49b not found: ID does not exist"
Oct 31 01:35:35.598206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.598159    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9" podStartSLOduration=1.598144738 podStartE2EDuration="1.598144738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:35.596144113 +0000 UTC m=+893.544792505" watchObservedRunningTime="2026-10-31 01:35:35.598144738 +0000 UTC m=+893.546793141"
Oct 31 01:35:35.611231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.611209    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"]
Oct 31 01:35:35.621368 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:35.621332    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7l4hn"]
Oct 31 01:35:36.170555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:36.170522    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987f6046-5693-4632-b4f5-3197514c733f" path="/var/lib/kubelet/pods/987f6046-5693-4632-b4f5-3197514c733f/volumes"
Oct 31 01:35:41.778501 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:35:41.778468    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:35:46.583432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:46.583397    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-46qd9"
Oct 31 01:35:51.922083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.922030    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"]
Oct 31 01:35:51.922540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.922503    5293 cpu_manager.go:414] "containerMap: removing container" podUID="987f6046-5693-4632-b4f5-3197514c733f" containerName="manager"
Oct 31 01:35:51.922540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.922522    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987f6046-5693-4632-b4f5-3197514c733f" containerName="manager"
Oct 31 01:35:51.922651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.922604    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="987f6046-5693-4632-b4f5-3197514c733f" containerName="manager"
Oct 31 01:35:51.927245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.927223    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:51.941876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.941849    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 01:35:51.942238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.942220    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wvwnn\""
Oct 31 01:35:51.942314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.942222    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 01:35:51.943625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.943605    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"]
Oct 31 01:35:51.979757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.979682    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:51.979757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.979725    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28q2\" (UniqueName: \"kubernetes.io/projected/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-kube-api-access-g28q2\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:51.979933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:51.979818    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:52.080498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.080464    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:52.080695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.080532    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:52.080695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.080563    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28q2\" (UniqueName: \"kubernetes.io/projected/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-kube-api-access-g28q2\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:52.085134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.081736    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:52.087472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.087442    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:52.093028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.093006    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28q2\" (UniqueName: \"kubernetes.io/projected/828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba-kube-api-access-g28q2\") pod \"kuadrant-console-plugin-66c4fb575d-8wgfx\" (UID: \"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:52.228389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.228298    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:35:52.231838 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.231812    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:52.242711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.242683    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:35:52.250560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.250537    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"
Oct 31 01:35:52.265614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.265580    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 01:35:52.282093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.282063    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:35:52.282243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.282224    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d61f77a9-2b32-463f-bc28-64e21cd43a7b-config-file\") pod \"limitador-limitador-7c9b87d574-786vs\" (UID: \"d61f77a9-2b32-463f-bc28-64e21cd43a7b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:52.282286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.282261    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzmb\" (UniqueName: \"kubernetes.io/projected/d61f77a9-2b32-463f-bc28-64e21cd43a7b-kube-api-access-4rzmb\") pod \"limitador-limitador-7c9b87d574-786vs\" (UID: \"d61f77a9-2b32-463f-bc28-64e21cd43a7b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:52.382854 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.382818    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzmb\" (UniqueName: \"kubernetes.io/projected/d61f77a9-2b32-463f-bc28-64e21cd43a7b-kube-api-access-4rzmb\") pod \"limitador-limitador-7c9b87d574-786vs\" (UID: \"d61f77a9-2b32-463f-bc28-64e21cd43a7b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:52.383033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.382921    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d61f77a9-2b32-463f-bc28-64e21cd43a7b-config-file\") pod \"limitador-limitador-7c9b87d574-786vs\" (UID: \"d61f77a9-2b32-463f-bc28-64e21cd43a7b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:52.383569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.383545    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d61f77a9-2b32-463f-bc28-64e21cd43a7b-config-file\") pod \"limitador-limitador-7c9b87d574-786vs\" (UID: \"d61f77a9-2b32-463f-bc28-64e21cd43a7b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:52.394888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.394857    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzmb\" (UniqueName: \"kubernetes.io/projected/d61f77a9-2b32-463f-bc28-64e21cd43a7b-kube-api-access-4rzmb\") pod \"limitador-limitador-7c9b87d574-786vs\" (UID: \"d61f77a9-2b32-463f-bc28-64e21cd43a7b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:52.529575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.529497    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx"]
Oct 31 01:35:52.533353 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:35:52.533331    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828a21ed_5fd5_40d8_9a6f_d2701ab2f5ba.slice/crio-69afc85d6b75be5c8f3902e15b9f467607bb3b5e2392d449ad052cc93de4ad2d WatchSource:0}: Error finding container 69afc85d6b75be5c8f3902e15b9f467607bb3b5e2392d449ad052cc93de4ad2d: Status 404 returned error can't find the container with id 69afc85d6b75be5c8f3902e15b9f467607bb3b5e2392d449ad052cc93de4ad2d
Oct 31 01:35:52.534578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.534557    5293 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:35:52.546058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.546024    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:52.641688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.641656    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx" event={"ID":"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba","Type":"ContainerStarted","Data":"69afc85d6b75be5c8f3902e15b9f467607bb3b5e2392d449ad052cc93de4ad2d"}
Oct 31 01:35:52.819402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:52.819330    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:35:52.823347 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:35:52.823322    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61f77a9_2b32_463f_bc28_64e21cd43a7b.slice/crio-8badba827bc52edff65b97381e4738dce6fe20c1264a05f39fec335d99b5920e WatchSource:0}: Error finding container 8badba827bc52edff65b97381e4738dce6fe20c1264a05f39fec335d99b5920e: Status 404 returned error can't find the container with id 8badba827bc52edff65b97381e4738dce6fe20c1264a05f39fec335d99b5920e
Oct 31 01:35:53.647424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:53.647391    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs" event={"ID":"d61f77a9-2b32-463f-bc28-64e21cd43a7b","Type":"ContainerStarted","Data":"8badba827bc52edff65b97381e4738dce6fe20c1264a05f39fec335d99b5920e"}
Oct 31 01:35:54.652338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:54.652305    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs" event={"ID":"d61f77a9-2b32-463f-bc28-64e21cd43a7b","Type":"ContainerStarted","Data":"ab3284533c882d85ec4570c182891c0883797a01a1e2c49714708ce48b6046c6"}
Oct 31 01:35:54.652718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:54.652429    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:35:54.670122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:35:54.670081    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs" podStartSLOduration=1.326321799 podStartE2EDuration="2.670068608s" totalImagesPullingTime="1.343746809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:54.668796965 +0000 UTC m=+912.617445357" watchObservedRunningTime="2026-10-31 01:35:54.670068608 +0000 UTC m=+912.618717000"
Oct 31 01:36:05.657576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:36:05.657548    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-786vs"
Oct 31 01:36:11.777994 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:36:11.777958    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:36:27.781472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:36:27.781432    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx" event={"ID":"828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba","Type":"ContainerStarted","Data":"a7a68b4dc2d5afa0aec633ab5b5ca4a3cdcd9a10352796dbb152177b54abd2c3"}
Oct 31 01:36:27.797374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:36:27.797330    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8wgfx" podStartSLOduration=2.295145449 podStartE2EDuration="36.797317867s" totalImagesPullingTime="34.502172418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:27.797120085 +0000 UTC m=+945.745768474" watchObservedRunningTime="2026-10-31 01:36:27.797317867 +0000 UTC m=+945.745966259"
Oct 31 01:36:41.776656 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:36:41.776624    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:37:09.519726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:37:09.519692    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:37:11.777158 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:37:11.777124    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:37:41.776967 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:37:41.776884    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:38:04.364024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.363988    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"]
Oct 31 01:38:04.367457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.367432    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:04.370372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.370355    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 01:38:04.370482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.370461    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-h5nk9\""
Oct 31 01:38:04.378738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.378717    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"]
Oct 31 01:38:04.411934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.411902    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6a4c1a-a546-4493-8003-6e4fbc1e84a3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cxqrw\" (UID: \"da6a4c1a-a546-4493-8003-6e4fbc1e84a3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:04.412078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.412064    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qvp\" (UniqueName: \"kubernetes.io/projected/da6a4c1a-a546-4493-8003-6e4fbc1e84a3-kube-api-access-68qvp\") pod \"payload-pre-processing-7b98bb678c-cxqrw\" (UID: \"da6a4c1a-a546-4493-8003-6e4fbc1e84a3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:04.512464 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.512433    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6a4c1a-a546-4493-8003-6e4fbc1e84a3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cxqrw\" (UID: \"da6a4c1a-a546-4493-8003-6e4fbc1e84a3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:04.512619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.512493    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qvp\" (UniqueName: \"kubernetes.io/projected/da6a4c1a-a546-4493-8003-6e4fbc1e84a3-kube-api-access-68qvp\") pod \"payload-pre-processing-7b98bb678c-cxqrw\" (UID: \"da6a4c1a-a546-4493-8003-6e4fbc1e84a3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:04.513213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.513193    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6a4c1a-a546-4493-8003-6e4fbc1e84a3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cxqrw\" (UID: \"da6a4c1a-a546-4493-8003-6e4fbc1e84a3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:04.523653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.523627    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qvp\" (UniqueName: \"kubernetes.io/projected/da6a4c1a-a546-4493-8003-6e4fbc1e84a3-kube-api-access-68qvp\") pod \"payload-pre-processing-7b98bb678c-cxqrw\" (UID: \"da6a4c1a-a546-4493-8003-6e4fbc1e84a3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:04.680400 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.680368    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:04.955776 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:04.955699    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"]
Oct 31 01:38:04.959170 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:38:04.959144    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6a4c1a_a546_4493_8003_6e4fbc1e84a3.slice/crio-a0bb387aca54569a95230e8f72a3ed19c3a432880c9514383abdb07096dc7617 WatchSource:0}: Error finding container a0bb387aca54569a95230e8f72a3ed19c3a432880c9514383abdb07096dc7617: Status 404 returned error can't find the container with id a0bb387aca54569a95230e8f72a3ed19c3a432880c9514383abdb07096dc7617
Oct 31 01:38:05.165196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:05.165167    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw" event={"ID":"da6a4c1a-a546-4493-8003-6e4fbc1e84a3","Type":"ContainerStarted","Data":"a0bb387aca54569a95230e8f72a3ed19c3a432880c9514383abdb07096dc7617"}
Oct 31 01:38:08.178558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:08.178524    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw" event={"ID":"da6a4c1a-a546-4493-8003-6e4fbc1e84a3","Type":"ContainerStarted","Data":"e2b276693249ca942bf6d990fdecef6a07877be9f0f84b35a489e6203e1b3812"}
Oct 31 01:38:08.178959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:08.178628    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:08.196239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:08.196199    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw" podStartSLOduration=2.027514067 podStartE2EDuration="4.196187164s" totalImagesPullingTime="2.168673097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:08.195958412 +0000 UTC m=+1046.144606814" watchObservedRunningTime="2026-10-31 01:38:08.196187164 +0000 UTC m=+1046.144835547"
Oct 31 01:38:11.777221 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:38:11.777187    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:38:19.185369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:19.185343    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxqrw"
Oct 31 01:38:34.446303 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.446269    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"]
Oct 31 01:38:34.450141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.450117    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.453338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.453315    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 01:38:34.453453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.453399    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 01:38:34.453453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.453407    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bg6sl\""
Oct 31 01:38:34.453453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.453407    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 01:38:34.461113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.461091    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"]
Oct 31 01:38:34.551110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.551077    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.551279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.551123    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e82e90-a1d5-453d-ac98-c37d03673798-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.551279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.551189    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.551371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.551320    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.551371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.551348    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8829h\" (UniqueName: \"kubernetes.io/projected/a1e82e90-a1d5-453d-ac98-c37d03673798-kube-api-access-8829h\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.551371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.551369    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.652681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.652648    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8829h\" (UniqueName: \"kubernetes.io/projected/a1e82e90-a1d5-453d-ac98-c37d03673798-kube-api-access-8829h\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.652681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.652680    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.652935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.652703    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.652935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.652842    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e82e90-a1d5-453d-ac98-c37d03673798-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.652935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.652885    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.653142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.653013    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.653142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.653084    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.653249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.653151    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.653404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.653383    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.655303 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.655277    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1e82e90-a1d5-453d-ac98-c37d03673798-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.655812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.655790    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e82e90-a1d5-453d-ac98-c37d03673798-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.664922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.664898    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8829h\" (UniqueName: \"kubernetes.io/projected/a1e82e90-a1d5-453d-ac98-c37d03673798-kube-api-access-8829h\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd\" (UID: \"a1e82e90-a1d5-453d-ac98-c37d03673798\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:34.763801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:34.763720    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:35.027027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:35.026951    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:38:35.044033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:35.044010    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"]
Oct 31 01:38:35.047467 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:38:35.047443    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e82e90_a1d5_453d_ac98_c37d03673798.slice/crio-4b6c8523a322104328afa26027e2a7b999c2217d73ebbd931de462921f93fefa WatchSource:0}: Error finding container 4b6c8523a322104328afa26027e2a7b999c2217d73ebbd931de462921f93fefa: Status 404 returned error can't find the container with id 4b6c8523a322104328afa26027e2a7b999c2217d73ebbd931de462921f93fefa
Oct 31 01:38:35.287305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:35.287224    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd" event={"ID":"a1e82e90-a1d5-453d-ac98-c37d03673798","Type":"ContainerStarted","Data":"4b6c8523a322104328afa26027e2a7b999c2217d73ebbd931de462921f93fefa"}
Oct 31 01:38:36.643746 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:36.643709    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:38:37.296165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:37.296131    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd" event={"ID":"a1e82e90-a1d5-453d-ac98-c37d03673798","Type":"ContainerStarted","Data":"4a7c74a85fc2fc821843d53965f9deae575ed362efbba7584874ca177ee7242e"}
Oct 31 01:38:37.318911 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:37.318850    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd" podStartSLOduration=1.578509691 podStartE2EDuration="3.318838117s" totalImagesPullingTime="1.740328426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:37.31712538 +0000 UTC m=+1075.265773771" watchObservedRunningTime="2026-10-31 01:38:37.318838117 +0000 UTC m=+1075.267486509"
Oct 31 01:38:37.763840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:37.763806    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:38.051921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.051842    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"]
Oct 31 01:38:38.055374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.055351    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.063069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.062733    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 01:38:38.067612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.067586    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"]
Oct 31 01:38:38.190109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.190078    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.190109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.190124    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.190327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.190141    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3760c544-14ca-4dfd-9527-5054edd99b12-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.190327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.190273    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.190327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.190312    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qnc\" (UniqueName: \"kubernetes.io/projected/3760c544-14ca-4dfd-9527-5054edd99b12-kube-api-access-n4qnc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.190434 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.190334    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.291604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.291569    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.291781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.291616    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.291781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.291654    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.291781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.291677    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3760c544-14ca-4dfd-9527-5054edd99b12-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.291781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.291749    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.291981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.291780    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qnc\" (UniqueName: \"kubernetes.io/projected/3760c544-14ca-4dfd-9527-5054edd99b12-kube-api-access-n4qnc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.292093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.292071    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.292093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.292081    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.292370 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.292338    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.294118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.294095    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3760c544-14ca-4dfd-9527-5054edd99b12-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.294469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.294449    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3760c544-14ca-4dfd-9527-5054edd99b12-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.303626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.303573    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qnc\" (UniqueName: \"kubernetes.io/projected/3760c544-14ca-4dfd-9527-5054edd99b12-kube-api-access-n4qnc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cfzj2\" (UID: \"3760c544-14ca-4dfd-9527-5054edd99b12\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.372569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.372539    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:38.636305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.636233    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"]
Oct 31 01:38:38.640657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.640634    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.646561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.646518    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 01:38:38.654128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.654099    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"]
Oct 31 01:38:38.654238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.654168    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"]
Oct 31 01:38:38.657367 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:38:38.657340    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3760c544_14ca_4dfd_9527_5054edd99b12.slice/crio-11118403a7487f558b8b46fb3e011300a5a02a959d69685447c81ff5cc731dfa WatchSource:0}: Error finding container 11118403a7487f558b8b46fb3e011300a5a02a959d69685447c81ff5cc731dfa: Status 404 returned error can't find the container with id 11118403a7487f558b8b46fb3e011300a5a02a959d69685447c81ff5cc731dfa
Oct 31 01:38:38.796358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.796327    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tt7c\" (UniqueName: \"kubernetes.io/projected/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-kube-api-access-6tt7c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.796752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.796388    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.796752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.796517    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.796752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.796565    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.796752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.796637    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.796752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.796721    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.897810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.897774    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tt7c\" (UniqueName: \"kubernetes.io/projected/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-kube-api-access-6tt7c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.897971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.897825    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.897971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.897947    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.898152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.897985    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.898152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.898010    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.898152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.898034    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.898370 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.898350    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.898429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.898381    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.898466 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.898423    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.900167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.900146    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.900476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.900459    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.909870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.909840    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tt7c\" (UniqueName: \"kubernetes.io/projected/173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b-kube-api-access-6tt7c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v\" (UID: \"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:38.957767 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:38.957738    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:39.263069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:39.262966    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"]
Oct 31 01:38:39.266000 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:38:39.265974    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173fa9c9_e8df_483f_be0b_dd7cb6ba6a7b.slice/crio-487b4d9b5bb0ea2d99ce3fbd0e58519d9b2beff67abe0235377435d7d3cb6a71 WatchSource:0}: Error finding container 487b4d9b5bb0ea2d99ce3fbd0e58519d9b2beff67abe0235377435d7d3cb6a71: Status 404 returned error can't find the container with id 487b4d9b5bb0ea2d99ce3fbd0e58519d9b2beff67abe0235377435d7d3cb6a71
Oct 31 01:38:39.306595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:39.306488    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2" event={"ID":"3760c544-14ca-4dfd-9527-5054edd99b12","Type":"ContainerStarted","Data":"61de79da9a58d4f7410affb35f86428d9d5eec5f40773e959f63a2762e191b57"}
Oct 31 01:38:39.307003 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:39.306975    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2" event={"ID":"3760c544-14ca-4dfd-9527-5054edd99b12","Type":"ContainerStarted","Data":"11118403a7487f558b8b46fb3e011300a5a02a959d69685447c81ff5cc731dfa"}
Oct 31 01:38:39.308358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:39.308330    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v" event={"ID":"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b","Type":"ContainerStarted","Data":"487b4d9b5bb0ea2d99ce3fbd0e58519d9b2beff67abe0235377435d7d3cb6a71"}
Oct 31 01:38:39.344774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:39.344712    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2" podStartSLOduration=1.064494383 podStartE2EDuration="1.344698119s" totalImagesPullingTime="280.203736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:39.343133096 +0000 UTC m=+1077.291781484" watchObservedRunningTime="2026-10-31 01:38:39.344698119 +0000 UTC m=+1077.293346510"
Oct 31 01:38:39.372675 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:39.372643    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:39.392068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:39.392026    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:38:40.314307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:40.314273    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v" event={"ID":"173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b","Type":"ContainerStarted","Data":"e7d29751526ab1cc4babeba5ac48f1bb36bb913e91d180afdca78f5d067e0119"}
Oct 31 01:38:40.334342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:40.334278    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v" podStartSLOduration=2.066375644 podStartE2EDuration="2.334261808s" totalImagesPullingTime="267.886164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:40.332953938 +0000 UTC m=+1078.281602329" watchObservedRunningTime="2026-10-31 01:38:40.334261808 +0000 UTC m=+1078.282910200"
Oct 31 01:38:40.958401 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:40.958359    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:41.776194 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:38:41.776156    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:38:43.331709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.331648    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"]
Oct 31 01:38:43.339662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.339636    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.345866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.345841    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"]
Oct 31 01:38:43.348264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.348245    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 01:38:43.435595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.435560    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhbd\" (UniqueName: \"kubernetes.io/projected/808265a9-b1bb-4c13-ba57-6d4aa0e91931-kube-api-access-mfhbd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.435793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.435610    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.435793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.435710    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.435793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.435777    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.435940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.435802    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.435940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.435918    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808265a9-b1bb-4c13-ba57-6d4aa0e91931-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.536870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.536830    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhbd\" (UniqueName: \"kubernetes.io/projected/808265a9-b1bb-4c13-ba57-6d4aa0e91931-kube-api-access-mfhbd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.536870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.536877    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.537141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.536899    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.537141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.536927    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.537141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.536944    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.537141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.536983    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808265a9-b1bb-4c13-ba57-6d4aa0e91931-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.537369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.537344    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.537369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.537357    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.537474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.537447    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.539420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.539397    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808265a9-b1bb-4c13-ba57-6d4aa0e91931-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.539581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.539565    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808265a9-b1bb-4c13-ba57-6d4aa0e91931-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.548736 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.548708    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhbd\" (UniqueName: \"kubernetes.io/projected/808265a9-b1bb-4c13-ba57-6d4aa0e91931-kube-api-access-mfhbd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw\" (UID: \"808265a9-b1bb-4c13-ba57-6d4aa0e91931\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.653643 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.653614    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:43.930674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.930625    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"]
Oct 31 01:38:43.933351 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:38:43.933326    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod808265a9_b1bb_4c13_ba57_6d4aa0e91931.slice/crio-3afa28badf33ec49090fec0f50c5017116183a5b88163345fcfc76c329ef4d07 WatchSource:0}: Error finding container 3afa28badf33ec49090fec0f50c5017116183a5b88163345fcfc76c329ef4d07: Status 404 returned error can't find the container with id 3afa28badf33ec49090fec0f50c5017116183a5b88163345fcfc76c329ef4d07
Oct 31 01:38:43.968465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:43.968440    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:38:44.334943 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:44.334901    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" event={"ID":"808265a9-b1bb-4c13-ba57-6d4aa0e91931","Type":"ContainerStarted","Data":"84fd5159455a25d1dafee0f8111ecc12d7bb01bb78cfec6f9c4430c80023a1cc"}
Oct 31 01:38:44.335336 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:44.334959    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" event={"ID":"808265a9-b1bb-4c13-ba57-6d4aa0e91931","Type":"ContainerStarted","Data":"3afa28badf33ec49090fec0f50c5017116183a5b88163345fcfc76c329ef4d07"}
Oct 31 01:38:44.356222 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:44.356171    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podStartSLOduration=1.079629767 podStartE2EDuration="1.356153181s" totalImagesPullingTime="276.523414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:44.354579208 +0000 UTC m=+1082.303227599" watchObservedRunningTime="2026-10-31 01:38:44.356153181 +0000 UTC m=+1082.304801577"
Oct 31 01:38:44.654655 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:44.654625    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:44.764097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:44.764066    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:44.776685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:44.776663    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:45.350625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:45.350599    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd"
Oct 31 01:38:48.372973 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:48.372944    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:48.385773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:48.385749    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:48.958115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:48.958079    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:48.970564 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:48.970537    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:49.367528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:49.367435    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2"
Oct 31 01:38:49.368128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:49.368111    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v"
Oct 31 01:38:53.654738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:53.654652    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:53.666831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:53.666807    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:54.388507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:54.388478    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:38:59.725424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.725383    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"]
Oct 31 01:38:59.728807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.728788    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.746294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.746267    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"]
Oct 31 01:38:59.823987 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.823953    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mqqdw\""
Oct 31 01:38:59.873961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.873922    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.873961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.873959    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.874182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.873982    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6qj\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-kube-api-access-tl6qj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.874182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.874135    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.874252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.874192    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.874252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.874230    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.874316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.874280    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.874316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.874308    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.874376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.874332    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975217    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975254    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975269    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975290    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6qj\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-kube-api-access-tl6qj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975320    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975381    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975408    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975458    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975499    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975662    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975807    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975843    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.975945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.975890    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.976208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.976189    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.977796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.977771    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.978010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.977987    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.986787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.986762    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6qj\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-kube-api-access-tl6qj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:38:59.986870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:38:59.986813    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x947t\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:39:00.043830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:00.043798    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:39:00.328111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:00.328018    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"]
Oct 31 01:39:00.332710 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:39:00.332677    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5860d1_93c7_4f7b_8c19_f08c0197e06e.slice/crio-6f0764e209fe72c1783c60bde7a63f0d6903062cc9499c842b20ad1956615f30 WatchSource:0}: Error finding container 6f0764e209fe72c1783c60bde7a63f0d6903062cc9499c842b20ad1956615f30: Status 404 returned error can't find the container with id 6f0764e209fe72c1783c60bde7a63f0d6903062cc9499c842b20ad1956615f30
Oct 31 01:39:00.405210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:00.405171    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" event={"ID":"0b5860d1-93c7-4f7b-8c19-f08c0197e06e","Type":"ContainerStarted","Data":"6f0764e209fe72c1783c60bde7a63f0d6903062cc9499c842b20ad1956615f30"}
Oct 31 01:39:04.007716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:04.007678    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:04.008119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:04.007792    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:04.008119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:04.007842    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:04.428519 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:04.428481    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" event={"ID":"0b5860d1-93c7-4f7b-8c19-f08c0197e06e","Type":"ContainerStarted","Data":"6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561"}
Oct 31 01:39:04.451645 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:04.451590    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podStartSLOduration=1.780119879 podStartE2EDuration="5.451572924s" totalImagesPullingTime="3.671453045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:04.450854875 +0000 UTC m=+1102.399503273" watchObservedRunningTime="2026-10-31 01:39:04.451572924 +0000 UTC m=+1102.400221316"
Oct 31 01:39:05.043988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:05.043950    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:39:05.045419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:05.045395    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x947t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 01:39:05.045542 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:05.045441    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 01:39:06.044531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:06.044491    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x947t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 01:39:06.044904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:06.044595    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 01:39:07.045059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:07.044993    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x947t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 01:39:07.045485 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:07.045094    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 01:39:08.044455 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:08.044419    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x947t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 01:39:08.044709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:08.044528    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 01:39:09.045215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:09.045181    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x947t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 01:39:09.045612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:09.045251    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 01:39:10.044671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:10.044633    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x947t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 01:39:10.044859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:10.044702    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 01:39:11.044146 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:11.044108    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x947t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 01:39:11.044525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:11.044158    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 01:39:11.777343 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:39:11.777301    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:39:12.191574 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:12.191541    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:39:12.192055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:12.191971    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:39:12.192735 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:12.192720    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 01:39:14.731983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.731946    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"]
Oct 31 01:39:14.737286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.737263    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.751679 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.751653    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"]
Oct 31 01:39:14.812272 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812241    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774c137c-c103-4d59-b525-37ad6df863e5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.812459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812282    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.812459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812299    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.812459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812412    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.812579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812472    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774c137c-c103-4d59-b525-37ad6df863e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.812579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812495    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.812579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812545    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.812669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812646    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774c137c-c103-4d59-b525-37ad6df863e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.812704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.812668    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjw6\" (UniqueName: \"kubernetes.io/projected/774c137c-c103-4d59-b525-37ad6df863e5-kube-api-access-kfjw6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913354    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913384    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913409    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913445    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774c137c-c103-4d59-b525-37ad6df863e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913473    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913538    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913576    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774c137c-c103-4d59-b525-37ad6df863e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913863 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913699    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjw6\" (UniqueName: \"kubernetes.io/projected/774c137c-c103-4d59-b525-37ad6df863e5-kube-api-access-kfjw6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913863 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913749    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774c137c-c103-4d59-b525-37ad6df863e5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.913863 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913816    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.914006 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.913978    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.914231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.914201    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.914319 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.914298    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.914724 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.914700    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774c137c-c103-4d59-b525-37ad6df863e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.915862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.915834    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774c137c-c103-4d59-b525-37ad6df863e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.916079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.916054    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774c137c-c103-4d59-b525-37ad6df863e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.946723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.946697    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774c137c-c103-4d59-b525-37ad6df863e5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:14.949513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:14.949490    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjw6\" (UniqueName: \"kubernetes.io/projected/774c137c-c103-4d59-b525-37ad6df863e5-kube-api-access-kfjw6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6dz48\" (UID: \"774c137c-c103-4d59-b525-37ad6df863e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:15.050023 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:15.049945    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:15.329388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:15.329309    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"]
Oct 31 01:39:15.333316 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:39:15.333287    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774c137c_c103_4d59_b525_37ad6df863e5.slice/crio-cd4bfc58ebc4b4de1fa1dcf5484951230e74ea41270a58e3d9212d69492fb841 WatchSource:0}: Error finding container cd4bfc58ebc4b4de1fa1dcf5484951230e74ea41270a58e3d9212d69492fb841: Status 404 returned error can't find the container with id cd4bfc58ebc4b4de1fa1dcf5484951230e74ea41270a58e3d9212d69492fb841
Oct 31 01:39:15.335351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:15.335321    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:15.335431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:15.335383    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:15.335431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:15.335410    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:15.483025 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:15.482987    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" event={"ID":"774c137c-c103-4d59-b525-37ad6df863e5","Type":"ContainerStarted","Data":"c4f350494d51d2b0a8421cb2fab2257faefc5de61963108173d99e5ef35856e5"}
Oct 31 01:39:15.483190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:15.483080    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" event={"ID":"774c137c-c103-4d59-b525-37ad6df863e5","Type":"ContainerStarted","Data":"cd4bfc58ebc4b4de1fa1dcf5484951230e74ea41270a58e3d9212d69492fb841"}
Oct 31 01:39:15.510594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:15.510534    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podStartSLOduration=1.5105159179999998 podStartE2EDuration="1.510515918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:15.509716599 +0000 UTC m=+1113.458364991" watchObservedRunningTime="2026-10-31 01:39:15.510515918 +0000 UTC m=+1113.459164311"
Oct 31 01:39:16.050383 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:16.050351    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:16.051823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:16.051802    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 01:39:16.051878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:16.051848    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 01:39:17.051256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:17.051225    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 01:39:17.051641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:17.051285    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 01:39:18.050337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:18.050298    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 01:39:18.050515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:18.050357    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 01:39:19.050714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:19.050678    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 01:39:19.051135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:19.050732    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 01:39:20.050414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:20.050381    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 01:39:20.050627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:20.050441    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 01:39:21.051199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:21.051162    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 01:39:21.051605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:21.051224    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 01:39:22.050878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:22.050843    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 01:39:22.051077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:22.050896    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 01:39:23.509300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:23.509262    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:23.509856 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:23.509776    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:23.510887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:23.510865    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48"
Oct 31 01:39:41.577358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.577325    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"]
Oct 31 01:39:41.587088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.587060    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.593449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.593423    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4bf7efc6-openshift-default-dockercfg-6h8xb\""
Oct 31 01:39:41.599176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.599135    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"]
Oct 31 01:39:41.647125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.647087    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fx47\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-kube-api-access-6fx47\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.647927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.647897    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-socket\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.648068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.647959    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefcbad9-d929-40d4-b7b2-19b85d54715e-istiod-ca-cert\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.648068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.647998    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-podinfo\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.648068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.648032    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-token\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.648230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.648091    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-credential-socket\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.648230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.648121    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-certs\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.648230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.648177    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-data\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.648230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.648205    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-envoy\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.696825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.696790    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"]
Oct 31 01:39:41.701529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.701511    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.708612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.708590    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6e7ff10a-openshift-default-dockercfg-pfprc\""
Oct 31 01:39:41.714392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.714369    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"]
Oct 31 01:39:41.749657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.749622    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lr5\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-kube-api-access-57lr5\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.749840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.749691    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-data\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.749840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.749777    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-envoy\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.749840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.749810    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-credential-socket\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.750078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.749879    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fx47\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-kube-api-access-6fx47\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.750078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.749902    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-data\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.750078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.749920    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-envoy\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.750078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.749995    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-socket\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.750337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.750109    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefcbad9-d929-40d4-b7b2-19b85d54715e-istiod-ca-cert\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.750415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.750357    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-data\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.750596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.750572    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-socket\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.750688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.750651    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-podinfo\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.750787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.750682    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-token\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.750990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.750968    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-socket\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.751393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751114    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-certs\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.751393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751150    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-podinfo\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.751393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751175    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-token\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.751393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751210    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-credential-socket\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.751393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751216    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefcbad9-d929-40d4-b7b2-19b85d54715e-istiod-ca-cert\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.751393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751230    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-certs\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.751393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751252    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d99105-add1-4e99-961b-f6a15d5f1a66-istiod-ca-cert\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.751775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751652    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-credential-socket\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.751775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.751742    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-certs\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.754518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.754494    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-envoy\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.755217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.755183    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-podinfo\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.763605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.763557    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-token\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.764449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.764424    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fx47\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-kube-api-access-6fx47\") pod \"e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") " pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:41.782374 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:39:41.782298    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:39:41.852028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.851951    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-data\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.851984    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-envoy\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852056    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-socket\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852080    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-certs\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852104    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-podinfo\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852119    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-token\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852154    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d99105-add1-4e99-961b-f6a15d5f1a66-istiod-ca-cert\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852185    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lr5\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-kube-api-access-57lr5\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852233    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-credential-socket\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852497    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-data\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852564    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-certs\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852644    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-socket\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.852757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.852741    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-credential-socket\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.853401 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.853375    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d99105-add1-4e99-961b-f6a15d5f1a66-istiod-ca-cert\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.854521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.854497    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-envoy\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.854809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.854789    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-podinfo\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.864897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.864874    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-token\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.865447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.865430    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lr5\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-kube-api-access-57lr5\") pod \"e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") " pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:41.907388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:41.907345    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:42.018073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.018003    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:42.059333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.059306    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"]
Oct 31 01:39:42.062945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.062920    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.076207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.076185    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-aba34286-gw-openshift-default-dockercfg-4xpp5\""
Oct 31 01:39:42.087541 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.087514    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"]
Oct 31 01:39:42.155787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.155753    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-token\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.156064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.155794    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh5tm\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-kube-api-access-zh5tm\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.156064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.155825    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-socket\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.156064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.155984    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-podinfo\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.156064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.156026    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-data\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.156261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.156085    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-certs\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.156261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.156221    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-credential-socket\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.156340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.156282    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istiod-ca-cert\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.156340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.156321    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-envoy\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.212197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.212077    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"]
Oct 31 01:39:42.215622 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:39:42.215598    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefcbad9_d929_40d4_b7b2_19b85d54715e.slice/crio-da560d0d589fefdd1f0ac1693e47385c1bd07f669b2de7e1bca5e7da3a1c1514 WatchSource:0}: Error finding container da560d0d589fefdd1f0ac1693e47385c1bd07f669b2de7e1bca5e7da3a1c1514: Status 404 returned error can't find the container with id da560d0d589fefdd1f0ac1693e47385c1bd07f669b2de7e1bca5e7da3a1c1514
Oct 31 01:39:42.217878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.217836    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.217974 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.217927    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.218054 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.217974    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266209    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-certs\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266329    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-credential-socket\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266433    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istiod-ca-cert\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266504    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-envoy\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266573    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-token\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266604    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh5tm\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-kube-api-access-zh5tm\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266637    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-socket\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266677    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-certs\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266694    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-podinfo\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.266746    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-data\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.267147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.267063    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-data\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.269868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.269829    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-envoy\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.271062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.270432    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-podinfo\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.271354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.271329    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-socket\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.272113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.272088    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istiod-ca-cert\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.272912 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.272884    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-credential-socket\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.284417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.284395    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-token\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.284693 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.284673    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh5tm\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-kube-api-access-zh5tm\") pod \"e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") " pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.310870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.310842    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"]
Oct 31 01:39:42.314937 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:39:42.314912    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d99105_add1_4e99_961b_f6a15d5f1a66.slice/crio-7c308165d74e2df47b8d0dbdad318438dedfbc956b8cac9c17b408d13378f1a5 WatchSource:0}: Error finding container 7c308165d74e2df47b8d0dbdad318438dedfbc956b8cac9c17b408d13378f1a5: Status 404 returned error can't find the container with id 7c308165d74e2df47b8d0dbdad318438dedfbc956b8cac9c17b408d13378f1a5
Oct 31 01:39:42.317608 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.317579    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.317676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.317656    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.317712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.317700    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.387740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.387712    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-aba34286-gw-openshift-default-dockercfg-4xpp5\""
Oct 31 01:39:42.396813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.396781    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:42.631292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.631249    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" event={"ID":"cefcbad9-d929-40d4-b7b2-19b85d54715e","Type":"ContainerStarted","Data":"cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f"}
Oct 31 01:39:42.631768 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.631333    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" event={"ID":"cefcbad9-d929-40d4-b7b2-19b85d54715e","Type":"ContainerStarted","Data":"da560d0d589fefdd1f0ac1693e47385c1bd07f669b2de7e1bca5e7da3a1c1514"}
Oct 31 01:39:42.632887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.632859    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" event={"ID":"e7d99105-add1-4e99-961b-f6a15d5f1a66","Type":"ContainerStarted","Data":"bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea"}
Oct 31 01:39:42.632990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.632912    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" event={"ID":"e7d99105-add1-4e99-961b-f6a15d5f1a66","Type":"ContainerStarted","Data":"7c308165d74e2df47b8d0dbdad318438dedfbc956b8cac9c17b408d13378f1a5"}
Oct 31 01:39:42.652927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.652857    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podStartSLOduration=1.65280776 podStartE2EDuration="1.65280776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:42.651830492 +0000 UTC m=+1140.600478886" watchObservedRunningTime="2026-10-31 01:39:42.65280776 +0000 UTC m=+1140.601456153"
Oct 31 01:39:42.674311 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.674226    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podStartSLOduration=1.6742138039999999 podStartE2EDuration="1.674213804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:42.672297959 +0000 UTC m=+1140.620946351" watchObservedRunningTime="2026-10-31 01:39:42.674213804 +0000 UTC m=+1140.622862196"
Oct 31 01:39:42.700860 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.700828    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"]
Oct 31 01:39:42.704812 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:39:42.704771    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fb67f2_c591_43be_9ca0_b4857f34f0f4.slice/crio-5aaffafe11fff43cd350a01c3b4d202e13748364d3f20cd54a4943f10eaca809 WatchSource:0}: Error finding container 5aaffafe11fff43cd350a01c3b4d202e13748364d3f20cd54a4943f10eaca809: Status 404 returned error can't find the container with id 5aaffafe11fff43cd350a01c3b4d202e13748364d3f20cd54a4943f10eaca809
Oct 31 01:39:42.707206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.707175    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.707303 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.707253    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.707364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.707298    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:42.908030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:42.908001    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:43.018924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.018833    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:43.024189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.024164    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:43.447130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.447094    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"]
Oct 31 01:39:43.453433 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.453405    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.462756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.462732    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6198e156-openshift-default-dockercfg-t4rfc\""
Oct 31 01:39:43.470879 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.470834    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"]
Oct 31 01:39:43.580062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580012    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7zz\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-kube-api-access-sk7zz\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.580237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580165    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-data\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.580237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580211    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-socket\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.580317 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580246    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istiod-ca-cert\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.580317 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580265    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-credential-socket\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.580380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580309    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-certs\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.580490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580474    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-podinfo\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.580524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580508    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-envoy\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.580559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.580529    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-token\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.677066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.675537    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" event={"ID":"85fb67f2-c591-43be-9ca0-b4857f34f0f4","Type":"ContainerStarted","Data":"590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25"}
Oct 31 01:39:43.677066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.675728    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" event={"ID":"85fb67f2-c591-43be-9ca0-b4857f34f0f4","Type":"ContainerStarted","Data":"5aaffafe11fff43cd350a01c3b4d202e13748364d3f20cd54a4943f10eaca809"}
Oct 31 01:39:43.678020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.677730    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:43.680323 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.680299    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:39:43.682517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682485    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-credential-socket\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.682674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682530    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-certs\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.682674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682594    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-podinfo\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.682674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682626    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-envoy\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.682674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682655    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-token\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.682925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682715    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7zz\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-kube-api-access-sk7zz\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.682925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682777    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-data\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.682925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682831    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-socket\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.682925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.682864    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istiod-ca-cert\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.683755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.683728    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-credential-socket\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.683897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.683737    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istiod-ca-cert\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.683984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.683966    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-socket\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.684392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.684367    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-certs\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.684501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.684430    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-data\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.688631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.687033    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-podinfo\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.689027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.689002    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-envoy\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.695273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.695247    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7zz\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-kube-api-access-sk7zz\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.695364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.695313    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-token\") pod \"e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") " pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.701650 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.701563    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podStartSLOduration=1.7015478320000001 podStartE2EDuration="1.701547832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:43.699008495 +0000 UTC m=+1141.647656887" watchObservedRunningTime="2026-10-31 01:39:43.701547832 +0000 UTC m=+1141.650196225"
Oct 31 01:39:43.778570 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.778533    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:43.913274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:43.913244    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:44.105680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.105589    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"]
Oct 31 01:39:44.109927 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:39:44.109887    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e792dd8_1b35_4b8b_a4cf_7ded278a4201.slice/crio-f4a4e7ffaf5730e805cb7ba01d678547609e6c67caf9c08f3ed3068fbcb2e62c WatchSource:0}: Error finding container f4a4e7ffaf5730e805cb7ba01d678547609e6c67caf9c08f3ed3068fbcb2e62c: Status 404 returned error can't find the container with id f4a4e7ffaf5730e805cb7ba01d678547609e6c67caf9c08f3ed3068fbcb2e62c
Oct 31 01:39:44.112515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.112479    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:44.112639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.112567    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:44.112639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.112613    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:39:44.396998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.396971    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:44.402691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.402663    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:44.681452 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.681367    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" event={"ID":"2e792dd8-1b35-4b8b-a4cf-7ded278a4201","Type":"ContainerStarted","Data":"b9b8c21ecd5ccc7931f45b7fbb278917b9e3f98a972a1630705b5895efe35ff4"}
Oct 31 01:39:44.681452 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.681422    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" event={"ID":"2e792dd8-1b35-4b8b-a4cf-7ded278a4201","Type":"ContainerStarted","Data":"f4a4e7ffaf5730e805cb7ba01d678547609e6c67caf9c08f3ed3068fbcb2e62c"}
Oct 31 01:39:44.681988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.681969    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:44.682077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.682012    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:44.682687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.682670    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:39:44.682853 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.682842    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:39:44.703147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.703095    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podStartSLOduration=1.703078494 podStartE2EDuration="1.703078494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:44.701678412 +0000 UTC m=+1142.650326804" watchObservedRunningTime="2026-10-31 01:39:44.703078494 +0000 UTC m=+1142.651726886"
Oct 31 01:39:44.779331 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:44.779307    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:45.785153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:45.785123    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:46.692369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:46.692335    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:46.693229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:46.693203    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:39:47.314616 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:47.314580    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:39:48.064149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.064120    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"]
Oct 31 01:39:48.068851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.068829    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:48.080559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.080536    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"]
Oct 31 01:39:48.138892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.126307    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6198e156\""
Oct 31 01:39:48.158866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.158820    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l\" (UID: \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\") " pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:48.159137 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.158880    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crstl\" (UniqueName: \"kubernetes.io/projected/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-kube-api-access-crstl\") pod \"payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l\" (UID: \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\") " pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:48.159412 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.159371    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6198e156-dockercfg-4mnb6\""
Oct 31 01:39:48.260150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.260108    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l\" (UID: \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\") " pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:48.260319 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.260177    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crstl\" (UniqueName: \"kubernetes.io/projected/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-kube-api-access-crstl\") pod \"payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l\" (UID: \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\") " pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:48.260789 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.260770    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l\" (UID: \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\") " pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:48.271738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.271709    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crstl\" (UniqueName: \"kubernetes.io/projected/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-kube-api-access-crstl\") pod \"payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l\" (UID: \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\") " pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:48.383591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.383512    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:48.683591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.683562    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"]
Oct 31 01:39:48.687108 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:39:48.687074    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b70e08_5ca6_4bd0_824b_8d2327824a1c.slice/crio-3e458aa6a3d567adb5e24af35b3a09a029bd6ee9f5f647d54e9c00e51bf2f075 WatchSource:0}: Error finding container 3e458aa6a3d567adb5e24af35b3a09a029bd6ee9f5f647d54e9c00e51bf2f075: Status 404 returned error can't find the container with id 3e458aa6a3d567adb5e24af35b3a09a029bd6ee9f5f647d54e9c00e51bf2f075
Oct 31 01:39:48.703256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:48.703225    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l" event={"ID":"c5b70e08-5ca6-4bd0-824b-8d2327824a1c","Type":"ContainerStarted","Data":"3e458aa6a3d567adb5e24af35b3a09a029bd6ee9f5f647d54e9c00e51bf2f075"}
Oct 31 01:39:49.723665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:49.723627    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l" event={"ID":"c5b70e08-5ca6-4bd0-824b-8d2327824a1c","Type":"ContainerStarted","Data":"e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02"}
Oct 31 01:39:49.724204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:49.723697    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:39:49.741288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:49.741234    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l" podStartSLOduration=1.74121473 podStartE2EDuration="1.74121473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:49.740877369 +0000 UTC m=+1147.689525761" watchObservedRunningTime="2026-10-31 01:39:49.74121473 +0000 UTC m=+1147.689863122"
Oct 31 01:39:55.546376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.516771    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"]
Oct 31 01:39:55.750065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.707700    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:55.827459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.755845    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4db52a49-dockercfg-gwqdq\""
Oct 31 01:39:55.827459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.776131    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4db52a49\""
Oct 31 01:39:55.827459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.798160    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"]
Oct 31 01:39:55.895125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.895067    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccd0537-14e4-445e-8b2c-9027797db6b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6\" (UID: \"8ccd0537-14e4-445e-8b2c-9027797db6b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:55.895125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.895117    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfzvq\" (UniqueName: \"kubernetes.io/projected/8ccd0537-14e4-445e-8b2c-9027797db6b6-kube-api-access-hfzvq\") pod \"payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6\" (UID: \"8ccd0537-14e4-445e-8b2c-9027797db6b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:55.999685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.999641    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccd0537-14e4-445e-8b2c-9027797db6b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6\" (UID: \"8ccd0537-14e4-445e-8b2c-9027797db6b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:55.999949 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:55.999929    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfzvq\" (UniqueName: \"kubernetes.io/projected/8ccd0537-14e4-445e-8b2c-9027797db6b6-kube-api-access-hfzvq\") pod \"payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6\" (UID: \"8ccd0537-14e4-445e-8b2c-9027797db6b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:56.001458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:56.001431    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccd0537-14e4-445e-8b2c-9027797db6b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6\" (UID: \"8ccd0537-14e4-445e-8b2c-9027797db6b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:56.037684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:56.037654    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfzvq\" (UniqueName: \"kubernetes.io/projected/8ccd0537-14e4-445e-8b2c-9027797db6b6-kube-api-access-hfzvq\") pod \"payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6\" (UID: \"8ccd0537-14e4-445e-8b2c-9027797db6b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:56.231539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:56.231229    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:56.979300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:56.978892    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"]
Oct 31 01:39:57.994063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:57.993645    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6" event={"ID":"8ccd0537-14e4-445e-8b2c-9027797db6b6","Type":"ContainerStarted","Data":"6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699"}
Oct 31 01:39:57.994063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:57.993733    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6" event={"ID":"8ccd0537-14e4-445e-8b2c-9027797db6b6","Type":"ContainerStarted","Data":"b5be350adf6e132506e048a851ae2260f3f376104e9e1fe8b6944f9e9ffbd1f1"}
Oct 31 01:39:57.994619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:57.994168    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:39:58.046093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:58.045336    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6" podStartSLOduration=3.045314566 podStartE2EDuration="3.045314566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:58.032226381 +0000 UTC m=+1155.980874774" watchObservedRunningTime="2026-10-31 01:39:58.045314566 +0000 UTC m=+1155.993962958"
Oct 31 01:39:58.066568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:58.066514    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:39:59.719775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:59.719236    5293 patch_prober.go:29] interesting pod/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:39:59.719775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:39:59.719298    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:40:00.032225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.029907    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"]
Oct 31 01:40:00.034589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.034562    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.042686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.042135    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:40:00.042686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.042252    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:40:00.047913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.047679    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"]
Oct 31 01:40:00.047913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.047786    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:40:00.047913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.047876    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:40:00.096668 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.096595    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.096844 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.096753    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.096844 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.096799    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f49rr\" (UniqueName: \"kubernetes.io/projected/87c3e1f8-bb07-403f-aab3-b1d92639f099-kube-api-access-f49rr\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.197516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.197468    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.197694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.197579    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.197694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.197612    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f49rr\" (UniqueName: \"kubernetes.io/projected/87c3e1f8-bb07-403f-aab3-b1d92639f099-kube-api-access-f49rr\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.198803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.198773    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.203811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.203785    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.212703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.212650    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f49rr\" (UniqueName: \"kubernetes.io/projected/87c3e1f8-bb07-403f-aab3-b1d92639f099-kube-api-access-f49rr\") pod \"maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.377386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.377286    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:00.392213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.392172    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"]
Oct 31 01:40:00.396590 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.396559    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.407067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.406965    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"]
Oct 31 01:40:00.503844 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.503788    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.503844 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.503838    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xkj\" (UniqueName: \"kubernetes.io/projected/1ae265e6-667a-441e-bc94-cdb884a45179-kube-api-access-n8xkj\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.504163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.503934    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.606214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.605423    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.606214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.605482    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xkj\" (UniqueName: \"kubernetes.io/projected/1ae265e6-667a-441e-bc94-cdb884a45179-kube-api-access-n8xkj\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.606214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.605614    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.607938 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.607889    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.610961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.610912    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.620428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.620371    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xkj\" (UniqueName: \"kubernetes.io/projected/1ae265e6-667a-441e-bc94-cdb884a45179-kube-api-access-n8xkj\") pod \"maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:00.727618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.727221    5293 patch_prober.go:29] interesting pod/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:40:00.727618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.727289    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:40:00.735238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.735189    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:40:00.750982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.750942    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"]
Oct 31 01:40:00.756496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:00.756471    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:01.049057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:01.048907    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69" event={"ID":"87c3e1f8-bb07-403f-aab3-b1d92639f099","Type":"ContainerStarted","Data":"9641a878ea5a5c8e6c0b400490e8645b0d470a17fa0e8d148d7cbedc1801a992"}
Oct 31 01:40:01.072273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:01.072235    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"]
Oct 31 01:40:01.077476 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:01.077442    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae265e6_667a_441e_bc94_cdb884a45179.slice/crio-38fe8dcb20add668da9c88369e81d25c2ad76bdf3174b1b93d2b17dea56c9a52 WatchSource:0}: Error finding container 38fe8dcb20add668da9c88369e81d25c2ad76bdf3174b1b93d2b17dea56c9a52: Status 404 returned error can't find the container with id 38fe8dcb20add668da9c88369e81d25c2ad76bdf3174b1b93d2b17dea56c9a52
Oct 31 01:40:02.061312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:02.061270    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n" event={"ID":"1ae265e6-667a-441e-bc94-cdb884a45179","Type":"ContainerStarted","Data":"38fe8dcb20add668da9c88369e81d25c2ad76bdf3174b1b93d2b17dea56c9a52"}
Oct 31 01:40:02.734278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:02.734234    5293 patch_prober.go:29] interesting pod/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:40:02.734481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:02.734304    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:40:04.602158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:04.602123    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:40:05.499134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.499099    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"]
Oct 31 01:40:05.539083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.510830    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.539083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.513827    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"]
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.571003    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.571169    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.571280    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtjff\" (UniqueName: \"kubernetes.io/projected/925e493d-4a87-4868-ae0e-582aeab5556e-kube-api-access-xtjff\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.671921    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.671976    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjff\" (UniqueName: \"kubernetes.io/projected/925e493d-4a87-4868-ae0e-582aeab5556e-kube-api-access-xtjff\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.672016    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.672681    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.674328    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.684094    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjff\" (UniqueName: \"kubernetes.io/projected/925e493d-4a87-4868-ae0e-582aeab5556e-kube-api-access-xtjff\") pod \"maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:05.882894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:05.845948    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:06.553516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:06.085176    5293 generic.go:356] "Generic (PLEG): container finished" podID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerID="bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea" exitCode=0
Oct 31 01:40:06.553516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:06.085252    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" event={"ID":"e7d99105-add1-4e99-961b-f6a15d5f1a66","Type":"ContainerDied","Data":"bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea"}
Oct 31 01:40:06.553516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:06.085645    5293 scope.go:122] "RemoveContainer" containerID="bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea"
Oct 31 01:40:06.553516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:06.086845    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:06.553516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:06.086914    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:06.553516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:06.086948    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:06.553516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:06.197632    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"]
Oct 31 01:40:07.092932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.092901    5293 generic.go:356] "Generic (PLEG): container finished" podID="1ae265e6-667a-441e-bc94-cdb884a45179" containerID="b7b790f2926611ce05d1f8eb71be4b51626121bfcce8ff2c4a44421ec63427dd" exitCode=0
Oct 31 01:40:07.093406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.092994    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n" event={"ID":"1ae265e6-667a-441e-bc94-cdb884a45179","Type":"ContainerDied","Data":"b7b790f2926611ce05d1f8eb71be4b51626121bfcce8ff2c4a44421ec63427dd"}
Oct 31 01:40:07.094957 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.094934    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" event={"ID":"e7d99105-add1-4e99-961b-f6a15d5f1a66","Type":"ContainerStarted","Data":"c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001"}
Oct 31 01:40:07.122132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.118769    5293 generic.go:356] "Generic (PLEG): container finished" podID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerID="590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25" exitCode=0
Oct 31 01:40:07.122132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.118879    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" event={"ID":"85fb67f2-c591-43be-9ca0-b4857f34f0f4","Type":"ContainerDied","Data":"590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25"}
Oct 31 01:40:07.122132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.119342    5293 scope.go:122] "RemoveContainer" containerID="590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25"
Oct 31 01:40:07.132118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.130715    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.132118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.130783    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.132118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.130816    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.154405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.154374    5293 generic.go:356] "Generic (PLEG): container finished" podID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerID="cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f" exitCode=0
Oct 31 01:40:07.154595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.154501    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" event={"ID":"cefcbad9-d929-40d4-b7b2-19b85d54715e","Type":"ContainerDied","Data":"cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f"}
Oct 31 01:40:07.154940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.154920    5293 scope.go:122] "RemoveContainer" containerID="cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f"
Oct 31 01:40:07.156782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.156738    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.156967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.156932    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.157151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.157130    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.157548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.157527    5293 generic.go:356] "Generic (PLEG): container finished" podID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerID="b9b8c21ecd5ccc7931f45b7fbb278917b9e3f98a972a1630705b5895efe35ff4" exitCode=0
Oct 31 01:40:07.157631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.157615    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" event={"ID":"2e792dd8-1b35-4b8b-a4cf-7ded278a4201","Type":"ContainerDied","Data":"b9b8c21ecd5ccc7931f45b7fbb278917b9e3f98a972a1630705b5895efe35ff4"}
Oct 31 01:40:07.157935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.157915    5293 scope.go:122] "RemoveContainer" containerID="b9b8c21ecd5ccc7931f45b7fbb278917b9e3f98a972a1630705b5895efe35ff4"
Oct 31 01:40:07.159462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.159426    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.159555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.159485    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.159555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.159515    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:07.163082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.161078    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz" event={"ID":"925e493d-4a87-4868-ae0e-582aeab5556e","Type":"ContainerStarted","Data":"fa9b7e9570a55073459d21a2e0236a4dc027ee15e16e176fd3c66306fbb8ee5f"}
Oct 31 01:40:07.163082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.161116    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz" event={"ID":"925e493d-4a87-4868-ae0e-582aeab5556e","Type":"ContainerStarted","Data":"c3f3b0b574834e38c66577df430743f6687a43737c940ef17f22267e6a5ff9b9"}
Oct 31 01:40:07.226276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:07.216200    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz" podStartSLOduration=2.216179304 podStartE2EDuration="2.216179304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:07.192720469 +0000 UTC m=+1165.141368861" watchObservedRunningTime="2026-10-31 01:40:07.216179304 +0000 UTC m=+1165.164827700"
Oct 31 01:40:08.018802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.018636    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:40:08.023956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.023916    5293 patch_prober.go:29] interesting pod/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:40:08.024286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.024243    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 01:40:08.269499 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.269402    5293 generic.go:356] "Generic (PLEG): container finished" podID="925e493d-4a87-4868-ae0e-582aeab5556e" containerID="fa9b7e9570a55073459d21a2e0236a4dc027ee15e16e176fd3c66306fbb8ee5f" exitCode=0
Oct 31 01:40:08.270205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.270169    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz" event={"ID":"925e493d-4a87-4868-ae0e-582aeab5556e","Type":"ContainerDied","Data":"fa9b7e9570a55073459d21a2e0236a4dc027ee15e16e176fd3c66306fbb8ee5f"}
Oct 31 01:40:08.275525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.275500    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" event={"ID":"85fb67f2-c591-43be-9ca0-b4857f34f0f4","Type":"ContainerStarted","Data":"8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1"}
Oct 31 01:40:08.309575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.309538    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" event={"ID":"cefcbad9-d929-40d4-b7b2-19b85d54715e","Type":"ContainerStarted","Data":"49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4"}
Oct 31 01:40:08.330067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.317125    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" event={"ID":"2e792dd8-1b35-4b8b-a4cf-7ded278a4201","Type":"ContainerStarted","Data":"427411b43236d79647eb9a4c19606a5da0b231fd487519be812855996548d801"}
Oct 31 01:40:08.417065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.404366    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:40:08.417065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.404498    5293 patch_prober.go:29] interesting pod/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:40:08.417065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.404540    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:40:08.801432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.786235    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:40:08.952149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.950226    5293 patch_prober.go:29] interesting pod/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:40:08.952149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.950290    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:40:08.952149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.950840    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:40:08.978072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.967184    5293 patch_prober.go:29] interesting pod/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:40:08.978072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:08.967251    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:40:09.022196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.022159    5293 patch_prober.go:29] interesting pod/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:40:09.022522 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.022490    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 01:40:09.154137 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.139302    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:40:09.418065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.405147    5293 patch_prober.go:29] interesting pod/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:40:09.484067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.451142    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:40:09.519292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.519259    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:09.604066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.594014    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n" event={"ID":"1ae265e6-667a-441e-bc94-cdb884a45179","Type":"ContainerDied","Data":"38fe8dcb20add668da9c88369e81d25c2ad76bdf3174b1b93d2b17dea56c9a52"}
Oct 31 01:40:09.634642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.617094    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38fe8dcb20add668da9c88369e81d25c2ad76bdf3174b1b93d2b17dea56c9a52"
Oct 31 01:40:09.634642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.617844    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-xks-ca\") pod \"1ae265e6-667a-441e-bc94-cdb884a45179\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") "
Oct 31 01:40:09.634642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.617901    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae265e6-667a-441e-bc94-cdb884a45179-kube-api-access-n8xkj\" (UniqueName: \"kubernetes.io/projected/1ae265e6-667a-441e-bc94-cdb884a45179-kube-api-access-n8xkj\") pod \"1ae265e6-667a-441e-bc94-cdb884a45179\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") "
Oct 31 01:40:09.634642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.617947    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-openshift-ca\") pod \"1ae265e6-667a-441e-bc94-cdb884a45179\" (UID: \"1ae265e6-667a-441e-bc94-cdb884a45179\") "
Oct 31 01:40:09.650281 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.650241    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-openshift-ca" pod "1ae265e6-667a-441e-bc94-cdb884a45179" (UID: "1ae265e6-667a-441e-bc94-cdb884a45179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:09.670565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.670486    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae265e6-667a-441e-bc94-cdb884a45179-kube-api-access-n8xkj" pod "1ae265e6-667a-441e-bc94-cdb884a45179" (UID: "1ae265e6-667a-441e-bc94-cdb884a45179"). InnerVolumeSpecName "kube-api-access-n8xkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:09.723972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.723940    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8xkj\" (UniqueName: \"kubernetes.io/projected/1ae265e6-667a-441e-bc94-cdb884a45179-kube-api-access-n8xkj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:09.739753 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.739722    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:09.845067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.783524    5293 patch_prober.go:29] interesting pod/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:40:09.900601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.900550    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:40:09.910094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.795732    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-xks-ca" pod "1ae265e6-667a-441e-bc94-cdb884a45179" (UID: "1ae265e6-667a-441e-bc94-cdb884a45179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:09.944316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.944234    5293 patch_prober.go:29] interesting pod/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:40:09.944559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.944535    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:40:09.989560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:09.989526    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae265e6-667a-441e-bc94-cdb884a45179-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:10.022927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.022435    5293 patch_prober.go:29] interesting pod/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:40:10.022927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.022508    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 01:40:10.409598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.407979    5293 patch_prober.go:29] interesting pod/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:40:10.409598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.408055    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:40:10.685932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.674104    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"
Oct 31 01:40:10.752234 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.752205    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:10.785342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.784639    5293 patch_prober.go:29] interesting pod/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:40:10.785342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.784723    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:40:10.922652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.922617    5293 patch_prober.go:29] interesting pod/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:40:10.923281 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.923250    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:40:10.973550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.973462    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-openshift-ca\") pod \"925e493d-4a87-4868-ae0e-582aeab5556e\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") "
Oct 31 01:40:10.974003 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.973981    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-xks-ca\") pod \"925e493d-4a87-4868-ae0e-582aeab5556e\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") "
Oct 31 01:40:10.974235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.974201    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925e493d-4a87-4868-ae0e-582aeab5556e-kube-api-access-xtjff\" (UniqueName: \"kubernetes.io/projected/925e493d-4a87-4868-ae0e-582aeab5556e-kube-api-access-xtjff\") pod \"925e493d-4a87-4868-ae0e-582aeab5556e\" (UID: \"925e493d-4a87-4868-ae0e-582aeab5556e\") "
Oct 31 01:40:10.976255 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:10.976232    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-openshift-ca" pod "925e493d-4a87-4868-ae0e-582aeab5556e" (UID: "925e493d-4a87-4868-ae0e-582aeab5556e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:11.014210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.014173    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-xks-ca" pod "925e493d-4a87-4868-ae0e-582aeab5556e" (UID: "925e493d-4a87-4868-ae0e-582aeab5556e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:11.020440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.020410    5293 patch_prober.go:29] interesting pod/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:40:11.020665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.020639    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 01:40:11.024824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.024780    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925e493d-4a87-4868-ae0e-582aeab5556e-kube-api-access-xtjff" pod "925e493d-4a87-4868-ae0e-582aeab5556e" (UID: "925e493d-4a87-4868-ae0e-582aeab5556e"). InnerVolumeSpecName "kube-api-access-xtjff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:11.082340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.082303    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:11.082590 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.082571    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtjff\" (UniqueName: \"kubernetes.io/projected/925e493d-4a87-4868-ae0e-582aeab5556e-kube-api-access-xtjff\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:11.082698 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.082686    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925e493d-4a87-4868-ae0e-582aeab5556e-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:11.398129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.397504    5293 patch_prober.go:29] interesting pod/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:40:11.398129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.397559    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:40:11.785711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.785610    5293 patch_prober.go:29] interesting pod/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:40:11.786338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.786287    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:40:11.902566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.902526    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz" event={"ID":"925e493d-4a87-4868-ae0e-582aeab5556e","Type":"ContainerDied","Data":"c3f3b0b574834e38c66577df430743f6687a43737c940ef17f22267e6a5ff9b9"}
Oct 31 01:40:11.911218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.911185    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3f3b0b574834e38c66577df430743f6687a43737c940ef17f22267e6a5ff9b9"
Oct 31 01:40:11.924067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.911530    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"
Oct 31 01:40:11.991080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.949566    5293 patch_prober.go:29] interesting pod/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:40:11.991080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:11.949662    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:40:12.097117 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:12.095301    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:40:12.097117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:12.095805    5293 patch_prober.go:29] interesting pod/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:40:12.097117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:12.095855    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 01:40:12.416126 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:12.400446    5293 patch_prober.go:29] interesting pod/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:40:12.416126 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:12.400501    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:40:12.782115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:12.781432    5293 patch_prober.go:29] interesting pod/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:40:12.782115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:12.781499    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:40:12.938067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:12.935914    5293 patch_prober.go:29] interesting pod/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:40:12.938067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:12.935973    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:40:13.021079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.020456    5293 patch_prober.go:29] interesting pod/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:40:13.021079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.020523    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 01:40:13.404406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.404371    5293 patch_prober.go:29] interesting pod/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:40:13.404695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.404666    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:40:13.681287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.681191    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:40:13.782066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.781502    5293 patch_prober.go:29] interesting pod/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:40:13.782066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.781561    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:40:13.918064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.917501    5293 patch_prober.go:29] interesting pod/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:40:13.918064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:13.917562    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:40:14.040528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.040391    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:40:14.052357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.052323    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:40:14.415470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.415436    5293 patch_prober.go:29] interesting pod/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:40:14.415729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.415700    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:40:14.682482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.682435    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:40:14.683969 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.683947    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:40:14.787441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.787412    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:40:14.788373 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.788313    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:40:14.790158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.790141    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:40:14.912627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.912598    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:40:14.964621 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.964543    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69" event={"ID":"87c3e1f8-bb07-403f-aab3-b1d92639f099","Type":"ContainerStarted","Data":"b43ed0c247e924dbb18e9ad4f248f75d5af12e7bfe177a6683b132765ab94ce1"}
Oct 31 01:40:14.965694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.965670    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:40:14.980989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:14.980945    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69" podStartSLOduration=1.646194827 podStartE2EDuration="14.980932429s" totalImagesPullingTime="13.334737602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:14.980785781 +0000 UTC m=+1172.929434174" watchObservedRunningTime="2026-10-31 01:40:14.980932429 +0000 UTC m=+1172.929580822"
Oct 31 01:40:15.401461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:15.401430    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:40:15.402556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:15.402536    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:40:15.971498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:15.971463    5293 generic.go:356] "Generic (PLEG): container finished" podID="87c3e1f8-bb07-403f-aab3-b1d92639f099" containerID="b43ed0c247e924dbb18e9ad4f248f75d5af12e7bfe177a6683b132765ab94ce1" exitCode=0
Oct 31 01:40:15.971691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:15.971554    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69" event={"ID":"87c3e1f8-bb07-403f-aab3-b1d92639f099","Type":"ContainerDied","Data":"b43ed0c247e924dbb18e9ad4f248f75d5af12e7bfe177a6683b132765ab94ce1"}
Oct 31 01:40:16.411282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:16.411248    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"]
Oct 31 01:40:16.417121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:16.417094    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e7ff10a-j7qpz"]
Oct 31 01:40:17.152883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.152843    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:17.295114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.295003    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c3e1f8-bb07-403f-aab3-b1d92639f099-kube-api-access-f49rr\" (UniqueName: \"kubernetes.io/projected/87c3e1f8-bb07-403f-aab3-b1d92639f099-kube-api-access-f49rr\") pod \"87c3e1f8-bb07-403f-aab3-b1d92639f099\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") "
Oct 31 01:40:17.295114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.295082    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-openshift-ca\") pod \"87c3e1f8-bb07-403f-aab3-b1d92639f099\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") "
Oct 31 01:40:17.295277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.295118    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-xks-ca\") pod \"87c3e1f8-bb07-403f-aab3-b1d92639f099\" (UID: \"87c3e1f8-bb07-403f-aab3-b1d92639f099\") "
Oct 31 01:40:17.295449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.295424    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-openshift-ca" pod "87c3e1f8-bb07-403f-aab3-b1d92639f099" (UID: "87c3e1f8-bb07-403f-aab3-b1d92639f099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:17.297448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.297417    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-xks-ca" pod "87c3e1f8-bb07-403f-aab3-b1d92639f099" (UID: "87c3e1f8-bb07-403f-aab3-b1d92639f099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:17.297587 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.297511    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c3e1f8-bb07-403f-aab3-b1d92639f099-kube-api-access-f49rr" pod "87c3e1f8-bb07-403f-aab3-b1d92639f099" (UID: "87c3e1f8-bb07-403f-aab3-b1d92639f099"). InnerVolumeSpecName "kube-api-access-f49rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:17.396593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.396556    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f49rr\" (UniqueName: \"kubernetes.io/projected/87c3e1f8-bb07-403f-aab3-b1d92639f099-kube-api-access-f49rr\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.396593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.396586    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.396763 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.396601    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c3e1f8-bb07-403f-aab3-b1d92639f099-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.982859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.982831    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"
Oct 31 01:40:17.983282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.982829    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69" event={"ID":"87c3e1f8-bb07-403f-aab3-b1d92639f099","Type":"ContainerDied","Data":"9641a878ea5a5c8e6c0b400490e8645b0d470a17fa0e8d148d7cbedc1801a992"}
Oct 31 01:40:17.983282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:17.982947    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9641a878ea5a5c8e6c0b400490e8645b0d470a17fa0e8d148d7cbedc1801a992"
Oct 31 01:40:18.067699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:18.067664    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"]
Oct 31 01:40:18.067860 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:18.067845    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy" containerID="cri-o://c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001" gracePeriod=30
Oct 31 01:40:18.198984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:18.198945    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925e493d-4a87-4868-ae0e-582aeab5556e" path="/var/lib/kubelet/pods/925e493d-4a87-4868-ae0e-582aeab5556e/volumes"
Oct 31 01:40:18.201718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:18.200986    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:40:19.147717 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.147684    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"]
Oct 31 01:40:19.348985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.348953    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"]
Oct 31 01:40:19.349447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349430    5293 cpu_manager.go:414] "containerMap: removing container" podUID="87c3e1f8-bb07-403f-aab3-b1d92639f099" containerName="revoke-keys"
Oct 31 01:40:19.349539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349450    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c3e1f8-bb07-403f-aab3-b1d92639f099" containerName="revoke-keys"
Oct 31 01:40:19.349539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349472    5293 cpu_manager.go:414] "containerMap: removing container" podUID="925e493d-4a87-4868-ae0e-582aeab5556e" containerName="revoke-keys"
Oct 31 01:40:19.349539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349480    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925e493d-4a87-4868-ae0e-582aeab5556e" containerName="revoke-keys"
Oct 31 01:40:19.349539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349502    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1ae265e6-667a-441e-bc94-cdb884a45179" containerName="revoke-keys"
Oct 31 01:40:19.349539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349510    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae265e6-667a-441e-bc94-cdb884a45179" containerName="revoke-keys"
Oct 31 01:40:19.349766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349608    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae265e6-667a-441e-bc94-cdb884a45179" containerName="revoke-keys"
Oct 31 01:40:19.349766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349627    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c3e1f8-bb07-403f-aab3-b1d92639f099" containerName="revoke-keys"
Oct 31 01:40:19.349766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.349638    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="925e493d-4a87-4868-ae0e-582aeab5556e" containerName="revoke-keys"
Oct 31 01:40:19.354078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.354031    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.367749 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.367720    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"]
Oct 31 01:40:19.374399 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.374371    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-be6935-openshift-default-dockercfg-vj9xz\""
Oct 31 01:40:19.422212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422128    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.422212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422181    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.422433 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422310    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.422433 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422359    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.422500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422478    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.422539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422505    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzc59\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-kube-api-access-fzc59\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.422586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422548    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177634d2-0f59-400b-8ad4-77188f8ce3cf-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.422654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422632    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.422687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.422667    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.523826 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.523795    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.523834    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.523868    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.523999    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.524121    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.524165    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.524242    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.524269    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzc59\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-kube-api-access-fzc59\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.524321    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177634d2-0f59-400b-8ad4-77188f8ce3cf-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.524407    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.524441    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.524832 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.524806    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.525158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.525103    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.526183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.525310    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177634d2-0f59-400b-8ad4-77188f8ce3cf-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.526840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.526822    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.527063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.527023    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.535621 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.535596    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.535782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.535762    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzc59\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-kube-api-access-fzc59\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.671278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.671248    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:19.954993 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.954966    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"]
Oct 31 01:40:19.957724 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:19.957698    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177634d2_0f59_400b_8ad4_77188f8ce3cf.slice/crio-82361b2cb109db4d34a2773cf4dcab53befdcfeaf6d92d28f01d6221fc8bfb63 WatchSource:0}: Error finding container 82361b2cb109db4d34a2773cf4dcab53befdcfeaf6d92d28f01d6221fc8bfb63: Status 404 returned error can't find the container with id 82361b2cb109db4d34a2773cf4dcab53befdcfeaf6d92d28f01d6221fc8bfb63
Oct 31 01:40:19.960103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.960072    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:19.960177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.960134    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:19.960177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.960161    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:19.993526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:19.993501    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" event={"ID":"177634d2-0f59-400b-8ad4-77188f8ce3cf","Type":"ContainerStarted","Data":"82361b2cb109db4d34a2773cf4dcab53befdcfeaf6d92d28f01d6221fc8bfb63"}
Oct 31 01:40:21.004222 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:21.004184    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" event={"ID":"177634d2-0f59-400b-8ad4-77188f8ce3cf","Type":"ContainerStarted","Data":"823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686"}
Oct 31 01:40:21.060750 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:21.036937    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podStartSLOduration=2.036915543 podStartE2EDuration="2.036915543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:21.032589121 +0000 UTC m=+1178.981237519" watchObservedRunningTime="2026-10-31 01:40:21.036915543 +0000 UTC m=+1178.985563937"
Oct 31 01:40:21.412347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:21.412310    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"]
Oct 31 01:40:21.414732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:21.414712    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bf7efc6-pgb69"]
Oct 31 01:40:21.595945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:21.595916    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"]
Oct 31 01:40:21.596166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:21.596140    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy" containerID="cri-o://49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4" gracePeriod=30
Oct 31 01:40:21.671629 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:21.671550    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:21.676873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:21.676850    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:22.009186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:22.009078    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:22.010521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:22.010504    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:22.173343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:22.173304    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c3e1f8-bb07-403f-aab3-b1d92639f099" path="/var/lib/kubelet/pods/87c3e1f8-bb07-403f-aab3-b1d92639f099/volumes"
Oct 31 01:40:22.409611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:22.409566    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"]
Oct 31 01:40:22.849696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:22.849615    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"]
Oct 31 01:40:22.853828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:22.853801    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aba34286-dbp6n"]
Oct 31 01:40:23.028539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.028474    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"]
Oct 31 01:40:23.029141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.028711    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy" containerID="cri-o://8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1" gracePeriod=30
Oct 31 01:40:23.326828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.326801    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:40:23.384704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.384665    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-podinfo\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.384883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.384744    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-socket\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.384883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.384818    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-envoy\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.384883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.384855    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-certs\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.385107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.384902    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-kube-api-access-57lr5\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-kube-api-access-57lr5\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.385107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.384933    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-data\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.385107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.384965    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-token\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-token\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.385286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.385122    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-credential-socket\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.385286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.385184    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d99105-add1-4e99-961b-f6a15d5f1a66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d99105-add1-4e99-961b-f6a15d5f1a66-istiod-ca-cert\") pod \"e7d99105-add1-4e99-961b-f6a15d5f1a66\" (UID: \"e7d99105-add1-4e99-961b-f6a15d5f1a66\") "
Oct 31 01:40:23.387239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.387190    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-socket" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:23.387469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.387437    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-credential-socket" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:23.387786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.387511    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-data" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:23.387786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.387563    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-certs" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:23.387786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.387714    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d99105-add1-4e99-961b-f6a15d5f1a66-istiod-ca-cert" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:23.392507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.392452    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-podinfo" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:40:23.392662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.392636    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-token" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:23.394075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.394023    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-envoy" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:23.394377 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.394323    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-kube-api-access-57lr5" pod "e7d99105-add1-4e99-961b-f6a15d5f1a66" (UID: "e7d99105-add1-4e99-961b-f6a15d5f1a66"). InnerVolumeSpecName "kube-api-access-57lr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:23.486907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486817    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:23.486907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486853    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:23.486907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486868    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57lr5\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-kube-api-access-57lr5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:23.486907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486883    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:23.486907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486897    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:23.487285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486913    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:23.487285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486929    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d99105-add1-4e99-961b-f6a15d5f1a66-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:23.487285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486942    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d99105-add1-4e99-961b-f6a15d5f1a66-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:23.487285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:23.486957    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d99105-add1-4e99-961b-f6a15d5f1a66-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:24.022333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.022297    5293 generic.go:356] "Generic (PLEG): container finished" podID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerID="c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001" exitCode=0
Oct 31 01:40:24.022569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.022360    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"
Oct 31 01:40:24.022569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.022397    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" event={"ID":"e7d99105-add1-4e99-961b-f6a15d5f1a66","Type":"ContainerDied","Data":"c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001"}
Oct 31 01:40:24.022569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.022464    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t" event={"ID":"e7d99105-add1-4e99-961b-f6a15d5f1a66","Type":"ContainerDied","Data":"7c308165d74e2df47b8d0dbdad318438dedfbc956b8cac9c17b408d13378f1a5"}
Oct 31 01:40:24.022569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.022498    5293 scope.go:122] "RemoveContainer" containerID="c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001"
Oct 31 01:40:24.037436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.037308    5293 scope.go:122] "RemoveContainer" containerID="bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea"
Oct 31 01:40:24.047443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.047421    5293 scope.go:122] "RemoveContainer" containerID="c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001"
Oct 31 01:40:24.047751 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:24.047732    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001\": container with ID starting with c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001 not found: ID does not exist" containerID="c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001"
Oct 31 01:40:24.047808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.047760    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001"} err="failed to get container status \"c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001\": rpc error: code = NotFound desc = could not find container \"c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001\": container with ID starting with c80745a86f5eca27d2c30f82f63d2baf343d2eeaaea2473105783210ba2ea001 not found: ID does not exist"
Oct 31 01:40:24.047808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.047778    5293 scope.go:122] "RemoveContainer" containerID="bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea"
Oct 31 01:40:24.048222 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:24.048205    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea\": container with ID starting with bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea not found: ID does not exist" containerID="bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea"
Oct 31 01:40:24.048290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.048231    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea"} err="failed to get container status \"bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea\": rpc error: code = NotFound desc = could not find container \"bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea\": container with ID starting with bfab19551d0544635ded67f0c7ad70f96bea524f54ccb536c35db8ab114a16ea not found: ID does not exist"
Oct 31 01:40:24.085398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.085366    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"]
Oct 31 01:40:24.125875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.125830    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6e7ff10a-openshift-default-5f8cb775f7-flk8t"]
Oct 31 01:40:24.148800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.148758    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6" podUID="8ccd0537-14e4-445e-8b2c-9027797db6b6" containerName="payload-pre-processing" containerID="cri-o://6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699" gracePeriod=25
Oct 31 01:40:24.175433 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.175399    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae265e6-667a-441e-bc94-cdb884a45179" path="/var/lib/kubelet/pods/1ae265e6-667a-441e-bc94-cdb884a45179/volumes"
Oct 31 01:40:24.176090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.176069    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" path="/var/lib/kubelet/pods/e7d99105-add1-4e99-961b-f6a15d5f1a66/volumes"
Oct 31 01:40:24.199859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.199832    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"]
Oct 31 01:40:24.200284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.200269    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy"
Oct 31 01:40:24.200355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.200285    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy"
Oct 31 01:40:24.200355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.200295    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy"
Oct 31 01:40:24.200355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.200300    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy"
Oct 31 01:40:24.200355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.200353    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy"
Oct 31 01:40:24.200504 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.200361    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d99105-add1-4e99-961b-f6a15d5f1a66" containerName="istio-proxy"
Oct 31 01:40:24.203916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.203894    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.212212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.212156    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0e8b5c-openshift-default-dockercfg-pq7tc\""
Oct 31 01:40:24.222431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.222407    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"]
Oct 31 01:40:24.295939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.295862    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.295939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.295899    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa358a0d-fd41-42a0-bbac-52875ea48c46-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.295939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.295921    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.296174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.296085    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.296174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.296118    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.296305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.296272    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.296380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.296310    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.296380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.296360    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-548tq\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-kube-api-access-548tq\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.296476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.296387    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397615    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397660    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397688    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-548tq\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-kube-api-access-548tq\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397721    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397807    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397837    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa358a0d-fd41-42a0-bbac-52875ea48c46-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397872    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397934    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.397973    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.398415    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.398852    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.399755    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa358a0d-fd41-42a0-bbac-52875ea48c46-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.399935    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.400152    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.401919    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.408068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.402107    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.419741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.419720    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.420633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.420617    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-548tq\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-kube-api-access-548tq\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.448425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.448401    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:40:24.498862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.498826    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccd0537-14e4-445e-8b2c-9027797db6b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccd0537-14e4-445e-8b2c-9027797db6b6-plugins-config-volume\") pod \"8ccd0537-14e4-445e-8b2c-9027797db6b6\" (UID: \"8ccd0537-14e4-445e-8b2c-9027797db6b6\") "
Oct 31 01:40:24.499082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.498887    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ccd0537-14e4-445e-8b2c-9027797db6b6-kube-api-access-hfzvq\" (UniqueName: \"kubernetes.io/projected/8ccd0537-14e4-445e-8b2c-9027797db6b6-kube-api-access-hfzvq\") pod \"8ccd0537-14e4-445e-8b2c-9027797db6b6\" (UID: \"8ccd0537-14e4-445e-8b2c-9027797db6b6\") "
Oct 31 01:40:24.499372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.499341    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccd0537-14e4-445e-8b2c-9027797db6b6-plugins-config-volume" pod "8ccd0537-14e4-445e-8b2c-9027797db6b6" (UID: "8ccd0537-14e4-445e-8b2c-9027797db6b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:24.501560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.501534    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccd0537-14e4-445e-8b2c-9027797db6b6-kube-api-access-hfzvq" pod "8ccd0537-14e4-445e-8b2c-9027797db6b6" (UID: "8ccd0537-14e4-445e-8b2c-9027797db6b6"). InnerVolumeSpecName "kube-api-access-hfzvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:24.519017 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.518988    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:24.600779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.600707    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfzvq\" (UniqueName: \"kubernetes.io/projected/8ccd0537-14e4-445e-8b2c-9027797db6b6-kube-api-access-hfzvq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:24.600779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.600745    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccd0537-14e4-445e-8b2c-9027797db6b6-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:24.815597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.815554    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"]
Oct 31 01:40:24.816411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.815802    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" containerID="cri-o://427411b43236d79647eb9a4c19606a5da0b231fd487519be812855996548d801" gracePeriod=30
Oct 31 01:40:24.851408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.851327    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"]
Oct 31 01:40:24.856971 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:24.856938    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa358a0d_fd41_42a0_bbac_52875ea48c46.slice/crio-3a1977b3c8e3af655c66708f82f0dbef8dd443e0805125addf151a9991832c08 WatchSource:0}: Error finding container 3a1977b3c8e3af655c66708f82f0dbef8dd443e0805125addf151a9991832c08: Status 404 returned error can't find the container with id 3a1977b3c8e3af655c66708f82f0dbef8dd443e0805125addf151a9991832c08
Oct 31 01:40:24.859496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.859458    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:24.859695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.859669    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:24.859835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:24.859817    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:25.032788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.032746    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" event={"ID":"fa358a0d-fd41-42a0-bbac-52875ea48c46","Type":"ContainerStarted","Data":"fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8"}
Oct 31 01:40:25.032973 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.032817    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" event={"ID":"fa358a0d-fd41-42a0-bbac-52875ea48c46","Type":"ContainerStarted","Data":"3a1977b3c8e3af655c66708f82f0dbef8dd443e0805125addf151a9991832c08"}
Oct 31 01:40:25.035796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.035769    5293 generic.go:356] "Generic (PLEG): container finished" podID="8ccd0537-14e4-445e-8b2c-9027797db6b6" containerID="6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699" exitCode=0
Oct 31 01:40:25.035926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.035815    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6" event={"ID":"8ccd0537-14e4-445e-8b2c-9027797db6b6","Type":"ContainerDied","Data":"6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699"}
Oct 31 01:40:25.035926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.035855    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6" event={"ID":"8ccd0537-14e4-445e-8b2c-9027797db6b6","Type":"ContainerDied","Data":"b5be350adf6e132506e048a851ae2260f3f376104e9e1fe8b6944f9e9ffbd1f1"}
Oct 31 01:40:25.035926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.035886    5293 scope.go:122] "RemoveContainer" containerID="6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699"
Oct 31 01:40:25.036107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.036006    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"
Oct 31 01:40:25.057731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.057420    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podStartSLOduration=1.057403467 podStartE2EDuration="1.057403467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:25.05378902 +0000 UTC m=+1183.002437433" watchObservedRunningTime="2026-10-31 01:40:25.057403467 +0000 UTC m=+1183.006051857"
Oct 31 01:40:25.060051 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.060016    5293 scope.go:122] "RemoveContainer" containerID="6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699"
Oct 31 01:40:25.060407 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:25.060381    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699\": container with ID starting with 6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699 not found: ID does not exist" containerID="6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699"
Oct 31 01:40:25.060470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.060418    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699"} err="failed to get container status \"6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699\": rpc error: code = NotFound desc = could not find container \"6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699\": container with ID starting with 6c6eb09e99f36112240aae5c840a0c40943e818b0ecc82dcdeb84845d75ee699 not found: ID does not exist"
Oct 31 01:40:25.068733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.068708    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"]
Oct 31 01:40:25.075369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.075330    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db52a49-8bf57c599-pl8g6"]
Oct 31 01:40:25.528112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.522306    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:25.538061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.538013    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:25.871861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:25.871781    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:40:26.042757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.042726    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:26.043831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.043813    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:26.175445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.175410    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ccd0537-14e4-445e-8b2c-9027797db6b6" path="/var/lib/kubelet/pods/8ccd0537-14e4-445e-8b2c-9027797db6b6/volumes"
Oct 31 01:40:26.855884 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.855861    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:40:26.961171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961140    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-kube-api-access-6fx47\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-kube-api-access-6fx47\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.961367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961180    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-credential-socket\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.961367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961221    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-token\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-token\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.961367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961240    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-socket\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.961367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961258    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-data\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.961367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961297    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-certs\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.961367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961335    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-podinfo\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.961367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961371    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-envoy\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.961729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.961415    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefcbad9-d929-40d4-b7b2-19b85d54715e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefcbad9-d929-40d4-b7b2-19b85d54715e-istiod-ca-cert\") pod \"cefcbad9-d929-40d4-b7b2-19b85d54715e\" (UID: \"cefcbad9-d929-40d4-b7b2-19b85d54715e\") "
Oct 31 01:40:26.962084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.962053    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-socket" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:26.962203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.962163    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-data" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:26.962266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.962195    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-credential-socket" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:26.962356 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.962334    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-certs" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:26.962356 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.962254    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefcbad9-d929-40d4-b7b2-19b85d54715e-istiod-ca-cert" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:26.964332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.964303    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-token" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:26.964463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.964379    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-kube-api-access-6fx47" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "kube-api-access-6fx47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:26.964781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.964766    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-podinfo" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:40:26.964834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:26.964790    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-envoy" pod "cefcbad9-d929-40d4-b7b2-19b85d54715e" (UID: "cefcbad9-d929-40d4-b7b2-19b85d54715e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:27.049653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.049617    5293 generic.go:356] "Generic (PLEG): container finished" podID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerID="49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4" exitCode=0
Oct 31 01:40:27.049813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.049666    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"
Oct 31 01:40:27.049813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.049695    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" event={"ID":"cefcbad9-d929-40d4-b7b2-19b85d54715e","Type":"ContainerDied","Data":"49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4"}
Oct 31 01:40:27.049813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.049770    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq" event={"ID":"cefcbad9-d929-40d4-b7b2-19b85d54715e","Type":"ContainerDied","Data":"da560d0d589fefdd1f0ac1693e47385c1bd07f669b2de7e1bca5e7da3a1c1514"}
Oct 31 01:40:27.049813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.049810    5293 scope.go:122] "RemoveContainer" containerID="49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4"
Oct 31 01:40:27.059732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.059712    5293 scope.go:122] "RemoveContainer" containerID="cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f"
Oct 31 01:40:27.062248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062218    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefcbad9-d929-40d4-b7b2-19b85d54715e-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.062248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062238    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fx47\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-kube-api-access-6fx47\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.062404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062253    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.062404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062267    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.062404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062275    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.062404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062284    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.062404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062292    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.062404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062300    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.062404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.062309    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefcbad9-d929-40d4-b7b2-19b85d54715e-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.068533 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.068504    5293 scope.go:122] "RemoveContainer" containerID="49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4"
Oct 31 01:40:27.068796 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:27.068778    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4\": container with ID starting with 49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4 not found: ID does not exist" containerID="49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4"
Oct 31 01:40:27.068861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.068804    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4"} err="failed to get container status \"49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4\": rpc error: code = NotFound desc = could not find container \"49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4\": container with ID starting with 49ef9f0d56d97937adaca9030b080aa4eb52caa1353fd8eb0c56a98ada2b11b4 not found: ID does not exist"
Oct 31 01:40:27.068861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.068819    5293 scope.go:122] "RemoveContainer" containerID="cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f"
Oct 31 01:40:27.069094 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:27.069033    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f\": container with ID starting with cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f not found: ID does not exist" containerID="cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f"
Oct 31 01:40:27.069142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.069095    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f"} err="failed to get container status \"cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f\": rpc error: code = NotFound desc = could not find container \"cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f\": container with ID starting with cadf56a6b04402d6d9fe68da3e7e86cf7756d8a907b7e98a9584eeb7f4cabf8f not found: ID does not exist"
Oct 31 01:40:27.073608 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.073587    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"]
Oct 31 01:40:27.078985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.078965    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4bf7efc6-openshift-default-b79fc4f4c-2hwzq"]
Oct 31 01:40:27.410620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.410582    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l" podUID="c5b70e08-5ca6-4bd0-824b-8d2327824a1c" containerName="payload-processing" containerID="cri-o://e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02" gracePeriod=25
Oct 31 01:40:27.686010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.685989    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:40:27.771508 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.771471    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-kube-api-access-crstl\" (UniqueName: \"kubernetes.io/projected/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-kube-api-access-crstl\") pod \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\" (UID: \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\") "
Oct 31 01:40:27.771707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.771520    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-plugins-config-volume\") pod \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\" (UID: \"c5b70e08-5ca6-4bd0-824b-8d2327824a1c\") "
Oct 31 01:40:27.772008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.771980    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-plugins-config-volume" pod "c5b70e08-5ca6-4bd0-824b-8d2327824a1c" (UID: "c5b70e08-5ca6-4bd0-824b-8d2327824a1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:27.773893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.773866    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-kube-api-access-crstl" pod "c5b70e08-5ca6-4bd0-824b-8d2327824a1c" (UID: "c5b70e08-5ca6-4bd0-824b-8d2327824a1c"). InnerVolumeSpecName "kube-api-access-crstl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:27.872966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.872934    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crstl\" (UniqueName: \"kubernetes.io/projected/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-kube-api-access-crstl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.872966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:27.872965    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b70e08-5ca6-4bd0-824b-8d2327824a1c-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.058325 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.058292    5293 generic.go:356] "Generic (PLEG): container finished" podID="c5b70e08-5ca6-4bd0-824b-8d2327824a1c" containerID="e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02" exitCode=0
Oct 31 01:40:28.058437 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.058347    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"
Oct 31 01:40:28.058437 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.058382    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l" event={"ID":"c5b70e08-5ca6-4bd0-824b-8d2327824a1c","Type":"ContainerDied","Data":"e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02"}
Oct 31 01:40:28.058541 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.058478    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l" event={"ID":"c5b70e08-5ca6-4bd0-824b-8d2327824a1c","Type":"ContainerDied","Data":"3e458aa6a3d567adb5e24af35b3a09a029bd6ee9f5f647d54e9c00e51bf2f075"}
Oct 31 01:40:28.058541 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.058510    5293 scope.go:122] "RemoveContainer" containerID="e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02"
Oct 31 01:40:28.128255 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.128223    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"]
Oct 31 01:40:28.133008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.132974    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6198e156-5456b9fd97-t2w7l"]
Oct 31 01:40:28.142636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.142618    5293 scope.go:122] "RemoveContainer" containerID="e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02"
Oct 31 01:40:28.143001 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:28.142978    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02\": container with ID starting with e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02 not found: ID does not exist" containerID="e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02"
Oct 31 01:40:28.143112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.143008    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02"} err="failed to get container status \"e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02\": rpc error: code = NotFound desc = could not find container \"e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02\": container with ID starting with e606d997bd3ca02d4be7cdd54c030c6c91bea7eae71f694a086fb9c79a872f02 not found: ID does not exist"
Oct 31 01:40:28.172203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.172173    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b70e08-5ca6-4bd0-824b-8d2327824a1c" path="/var/lib/kubelet/pods/c5b70e08-5ca6-4bd0-824b-8d2327824a1c/volumes"
Oct 31 01:40:28.172731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.172710    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" path="/var/lib/kubelet/pods/cefcbad9-d929-40d4-b7b2-19b85d54715e/volumes"
Oct 31 01:40:28.282065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.282017    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:40:28.317674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.317598    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"]
Oct 31 01:40:28.318089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318071    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8ccd0537-14e4-445e-8b2c-9027797db6b6" containerName="payload-pre-processing"
Oct 31 01:40:28.318180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318090    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccd0537-14e4-445e-8b2c-9027797db6b6" containerName="payload-pre-processing"
Oct 31 01:40:28.318180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318102    5293 cpu_manager.go:414] "containerMap: removing container" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy"
Oct 31 01:40:28.318180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318109    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy"
Oct 31 01:40:28.318180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318122    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c5b70e08-5ca6-4bd0-824b-8d2327824a1c" containerName="payload-processing"
Oct 31 01:40:28.318180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318130    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b70e08-5ca6-4bd0-824b-8d2327824a1c" containerName="payload-processing"
Oct 31 01:40:28.318180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318146    5293 cpu_manager.go:414] "containerMap: removing container" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy"
Oct 31 01:40:28.318180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318153    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy"
Oct 31 01:40:28.318506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318256    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy"
Oct 31 01:40:28.318506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318269    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy"
Oct 31 01:40:28.318506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318283    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccd0537-14e4-445e-8b2c-9027797db6b6" containerName="payload-pre-processing"
Oct 31 01:40:28.318506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318292    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy"
Oct 31 01:40:28.318506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.318301    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b70e08-5ca6-4bd0-824b-8d2327824a1c" containerName="payload-processing"
Oct 31 01:40:28.321350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.321331    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:28.327494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.327469    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-6ae519\""
Oct 31 01:40:28.333393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.333367    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"]
Oct 31 01:40:28.346634 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.346610    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"]
Oct 31 01:40:28.347300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.347155    5293 cpu_manager.go:414] "containerMap: removing container" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy"
Oct 31 01:40:28.347300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.347175    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerName="istio-proxy"
Oct 31 01:40:28.347300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.347196    5293 cpu_manager.go:414] "containerMap: removing container" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy"
Oct 31 01:40:28.347300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.347204    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy"
Oct 31 01:40:28.347557 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.347347    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefcbad9-d929-40d4-b7b2-19b85d54715e" containerName="istio-proxy"
Oct 31 01:40:28.350422 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.350405    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:28.356802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.356786    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-6ae519-dockercfg-frpjv\""
Oct 31 01:40:28.363732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.362205    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"]
Oct 31 01:40:28.378550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378528    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-token\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-token\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.378664 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378595    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-certs\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.378715 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378680    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-data\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.378775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378759    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-envoy\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.378833 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378789    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-socket\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.378833 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378826    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-kube-api-access-zh5tm\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-kube-api-access-zh5tm\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.378937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378851    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istiod-ca-cert\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.378937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378875    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-credential-socket\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.378937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.378904    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-podinfo\") pod \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\" (UID: \"85fb67f2-c591-43be-9ca0-b4857f34f0f4\") "
Oct 31 01:40:28.379111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379004    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt\" (UID: \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:28.379111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379103    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpgtr\" (UniqueName: \"kubernetes.io/projected/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-kube-api-access-hpgtr\") pod \"payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt\" (UID: \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:28.379204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379179    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-socket" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.379256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379212    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-data" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.379440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379417    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-certs" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.379526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379506    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istiod-ca-cert" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:28.379757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379729    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-credential-socket" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.379989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379963    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.379989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.379984    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.380099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.380000    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.380099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.380014    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.380099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.380027    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.381885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.381848    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-token" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:28.382461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.382431    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-kube-api-access-zh5tm" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "kube-api-access-zh5tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:28.382556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.382433    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-podinfo" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:40:28.383523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.383499    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-envoy" pod "85fb67f2-c591-43be-9ca0-b4857f34f0f4" (UID: "85fb67f2-c591-43be-9ca0-b4857f34f0f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.481035    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt\" (UID: \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.481131    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfpl\" (UniqueName: \"kubernetes.io/projected/082b7243-443f-49c1-90ad-955cbe31ec7f-kube-api-access-wkfpl\") pod \"payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h\" (UID: \"082b7243-443f-49c1-90ad-955cbe31ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.481163    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpgtr\" (UniqueName: \"kubernetes.io/projected/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-kube-api-access-hpgtr\") pod \"payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt\" (UID: \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.481268    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082b7243-443f-49c1-90ad-955cbe31ec7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h\" (UID: \"082b7243-443f-49c1-90ad-955cbe31ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.481329    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.481344    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh5tm\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-kube-api-access-zh5tm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.481359    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.481374    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85fb67f2-c591-43be-9ca0-b4857f34f0f4-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.483742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.482282    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt\" (UID: \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:28.495634 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.495609    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpgtr\" (UniqueName: \"kubernetes.io/projected/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-kube-api-access-hpgtr\") pod \"payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt\" (UID: \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:28.582101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.581979    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfpl\" (UniqueName: \"kubernetes.io/projected/082b7243-443f-49c1-90ad-955cbe31ec7f-kube-api-access-wkfpl\") pod \"payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h\" (UID: \"082b7243-443f-49c1-90ad-955cbe31ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:28.582101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.582077    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082b7243-443f-49c1-90ad-955cbe31ec7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h\" (UID: \"082b7243-443f-49c1-90ad-955cbe31ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:28.582713 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.582693    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082b7243-443f-49c1-90ad-955cbe31ec7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h\" (UID: \"082b7243-443f-49c1-90ad-955cbe31ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:28.594663 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.594638    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfpl\" (UniqueName: \"kubernetes.io/projected/082b7243-443f-49c1-90ad-955cbe31ec7f-kube-api-access-wkfpl\") pod \"payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h\" (UID: \"082b7243-443f-49c1-90ad-955cbe31ec7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:28.640110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.640081    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:28.666223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.666199    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:28.922248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.922105    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"]
Oct 31 01:40:28.924640 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:28.924610    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb196ab_0bf0_4a69_9b1f_61328e7a2804.slice/crio-d8e7cff5f326f91c9cc37b1e615988f9f409cf9b156a78e72bb93df9c084948f WatchSource:0}: Error finding container d8e7cff5f326f91c9cc37b1e615988f9f409cf9b156a78e72bb93df9c084948f: Status 404 returned error can't find the container with id d8e7cff5f326f91c9cc37b1e615988f9f409cf9b156a78e72bb93df9c084948f
Oct 31 01:40:28.984165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:28.984123    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"]
Oct 31 01:40:28.987592 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:28.987567    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082b7243_443f_49c1_90ad_955cbe31ec7f.slice/crio-645a41c3b06de941bb1818b66739fa093277937d5a25ac97255578f93088b17c WatchSource:0}: Error finding container 645a41c3b06de941bb1818b66739fa093277937d5a25ac97255578f93088b17c: Status 404 returned error can't find the container with id 645a41c3b06de941bb1818b66739fa093277937d5a25ac97255578f93088b17c
Oct 31 01:40:29.010999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.010973    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:40:29.067927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.067449    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt" event={"ID":"3cb196ab-0bf0-4a69-9b1f-61328e7a2804","Type":"ContainerStarted","Data":"5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8"}
Oct 31 01:40:29.067927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.067502    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt" event={"ID":"3cb196ab-0bf0-4a69-9b1f-61328e7a2804","Type":"ContainerStarted","Data":"d8e7cff5f326f91c9cc37b1e615988f9f409cf9b156a78e72bb93df9c084948f"}
Oct 31 01:40:29.067927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.067635    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:29.070087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.070061    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h" event={"ID":"082b7243-443f-49c1-90ad-955cbe31ec7f","Type":"ContainerStarted","Data":"b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2"}
Oct 31 01:40:29.070196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.070110    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h" event={"ID":"082b7243-443f-49c1-90ad-955cbe31ec7f","Type":"ContainerStarted","Data":"645a41c3b06de941bb1818b66739fa093277937d5a25ac97255578f93088b17c"}
Oct 31 01:40:29.070196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.070169    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:29.076495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.076397    5293 generic.go:356] "Generic (PLEG): container finished" podID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" containerID="8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1" exitCode=0
Oct 31 01:40:29.076495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.076450    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" event={"ID":"85fb67f2-c591-43be-9ca0-b4857f34f0f4","Type":"ContainerDied","Data":"8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1"}
Oct 31 01:40:29.076495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.076490    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2" event={"ID":"85fb67f2-c591-43be-9ca0-b4857f34f0f4","Type":"ContainerDied","Data":"5aaffafe11fff43cd350a01c3b4d202e13748364d3f20cd54a4943f10eaca809"}
Oct 31 01:40:29.076728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.076520    5293 scope.go:122] "RemoveContainer" containerID="8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1"
Oct 31 01:40:29.076728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.076623    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"
Oct 31 01:40:29.093903 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.093643    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt" podStartSLOduration=1.093625097 podStartE2EDuration="1.093625097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:29.091527994 +0000 UTC m=+1187.040176386" watchObservedRunningTime="2026-10-31 01:40:29.093625097 +0000 UTC m=+1187.042273490"
Oct 31 01:40:29.097257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.097231    5293 scope.go:122] "RemoveContainer" containerID="590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25"
Oct 31 01:40:29.109559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.109526    5293 scope.go:122] "RemoveContainer" containerID="8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1"
Oct 31 01:40:29.109872 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:29.109847    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1\": container with ID starting with 8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1 not found: ID does not exist" containerID="8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1"
Oct 31 01:40:29.109950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.109883    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1"} err="failed to get container status \"8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1\": rpc error: code = NotFound desc = could not find container \"8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1\": container with ID starting with 8c2774d8c62f472ad6d77d22f8b9e0d08bad569319207a2e943a896f8d2a31c1 not found: ID does not exist"
Oct 31 01:40:29.109950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.109911    5293 scope.go:122] "RemoveContainer" containerID="590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25"
Oct 31 01:40:29.110204 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:29.110183    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25\": container with ID starting with 590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25 not found: ID does not exist" containerID="590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25"
Oct 31 01:40:29.110264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.110215    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25"} err="failed to get container status \"590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25\": rpc error: code = NotFound desc = could not find container \"590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25\": container with ID starting with 590735de2fb60670af04bcea4497fc35b70e1e26f29ec2119b7a4f7716471a25 not found: ID does not exist"
Oct 31 01:40:29.127840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.127790    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h" podStartSLOduration=1.127749563 podStartE2EDuration="1.127749563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:29.126512947 +0000 UTC m=+1187.075161338" watchObservedRunningTime="2026-10-31 01:40:29.127749563 +0000 UTC m=+1187.076397956"
Oct 31 01:40:29.156099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.156058    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"]
Oct 31 01:40:29.157259 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.157240    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"]
Oct 31 01:40:29.161946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.161924    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:29.168586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.168565    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-aba34286-gw-openshift-default-68f4b54c77-7p2z2"]
Oct 31 01:40:29.169144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.169126    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-0e8b5c-dockercfg-w5c6p\""
Oct 31 01:40:29.174920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.174874    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-0e8b5c\""
Oct 31 01:40:29.179238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.179220    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"]
Oct 31 01:40:29.191621 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.191598    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"]
Oct 31 01:40:29.195109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.195092    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:29.210396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.210375    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"]
Oct 31 01:40:29.291127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.290297    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2cw8\" (UniqueName: \"kubernetes.io/projected/f22ea351-d8b6-499f-98ed-068aa33808d0-kube-api-access-v2cw8\") pod \"payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7\" (UID: \"f22ea351-d8b6-499f-98ed-068aa33808d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:29.291127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.290457    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2fk6\" (UniqueName: \"kubernetes.io/projected/ee13e9ec-c271-4076-9849-a843994efad4-kube-api-access-s2fk6\") pod \"payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw\" (UID: \"ee13e9ec-c271-4076-9849-a843994efad4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:29.291127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.290491    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22ea351-d8b6-499f-98ed-068aa33808d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7\" (UID: \"f22ea351-d8b6-499f-98ed-068aa33808d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:29.291127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.290555    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee13e9ec-c271-4076-9849-a843994efad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw\" (UID: \"ee13e9ec-c271-4076-9849-a843994efad4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:29.391200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.391165    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2cw8\" (UniqueName: \"kubernetes.io/projected/f22ea351-d8b6-499f-98ed-068aa33808d0-kube-api-access-v2cw8\") pod \"payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7\" (UID: \"f22ea351-d8b6-499f-98ed-068aa33808d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:29.391369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.391277    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2fk6\" (UniqueName: \"kubernetes.io/projected/ee13e9ec-c271-4076-9849-a843994efad4-kube-api-access-s2fk6\") pod \"payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw\" (UID: \"ee13e9ec-c271-4076-9849-a843994efad4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:29.391369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.391306    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22ea351-d8b6-499f-98ed-068aa33808d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7\" (UID: \"f22ea351-d8b6-499f-98ed-068aa33808d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:29.391484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.391363    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee13e9ec-c271-4076-9849-a843994efad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw\" (UID: \"ee13e9ec-c271-4076-9849-a843994efad4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:29.392155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.392131    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22ea351-d8b6-499f-98ed-068aa33808d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7\" (UID: \"f22ea351-d8b6-499f-98ed-068aa33808d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:29.392155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.392148    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee13e9ec-c271-4076-9849-a843994efad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw\" (UID: \"ee13e9ec-c271-4076-9849-a843994efad4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:29.405663 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.405634    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2cw8\" (UniqueName: \"kubernetes.io/projected/f22ea351-d8b6-499f-98ed-068aa33808d0-kube-api-access-v2cw8\") pod \"payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7\" (UID: \"f22ea351-d8b6-499f-98ed-068aa33808d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:29.406777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.406756    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2fk6\" (UniqueName: \"kubernetes.io/projected/ee13e9ec-c271-4076-9849-a843994efad4-kube-api-access-s2fk6\") pod \"payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw\" (UID: \"ee13e9ec-c271-4076-9849-a843994efad4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:29.475804 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.475721    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:29.512329 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.512286    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:29.791294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.791188    5293 patch_prober.go:29] interesting pod/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:40:29.791294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.791277    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:40:29.815061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.814503    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"]
Oct 31 01:40:29.840668 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:29.840636    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22ea351_d8b6_499f_98ed_068aa33808d0.slice/crio-55474a2228d8c009871802bf24bccd4f9ac153a2145d7a464e3bcca821fb0e0e WatchSource:0}: Error finding container 55474a2228d8c009871802bf24bccd4f9ac153a2145d7a464e3bcca821fb0e0e: Status 404 returned error can't find the container with id 55474a2228d8c009871802bf24bccd4f9ac153a2145d7a464e3bcca821fb0e0e
Oct 31 01:40:29.844765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:29.844403    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"]
Oct 31 01:40:30.082928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.082570    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"]
Oct 31 01:40:30.087410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.087371    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:30.091273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.090735    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw" event={"ID":"ee13e9ec-c271-4076-9849-a843994efad4","Type":"ContainerStarted","Data":"1b4832e30e281477cec70d5ec984ec191ce57caa05a2799043513a97d8fe8356"}
Oct 31 01:40:30.091273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.090775    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw" event={"ID":"ee13e9ec-c271-4076-9849-a843994efad4","Type":"ContainerStarted","Data":"b854f53ea89447668d22197ec132cddc57fdf3ff5a1bd84e86da40823e0a369a"}
Oct 31 01:40:30.094179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.093641    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7" event={"ID":"f22ea351-d8b6-499f-98ed-068aa33808d0","Type":"ContainerStarted","Data":"a22ec7223afc8e7954a9b60b6c80949e5d7aee8c041455fc7a54c3da028cbb93"}
Oct 31 01:40:30.094179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.093683    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7" event={"ID":"f22ea351-d8b6-499f-98ed-068aa33808d0","Type":"ContainerStarted","Data":"55474a2228d8c009871802bf24bccd4f9ac153a2145d7a464e3bcca821fb0e0e"}
Oct 31 01:40:30.094179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.093946    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"]
Oct 31 01:40:30.094179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.094135    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:30.125086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.100296    5293 generic.go:356] "Generic (PLEG): container finished" podID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerID="427411b43236d79647eb9a4c19606a5da0b231fd487519be812855996548d801" exitCode=0
Oct 31 01:40:30.125086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.100922    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" event={"ID":"2e792dd8-1b35-4b8b-a4cf-7ded278a4201","Type":"ContainerDied","Data":"427411b43236d79647eb9a4c19606a5da0b231fd487519be812855996548d801"}
Oct 31 01:40:30.125086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.100971    5293 scope.go:122] "RemoveContainer" containerID="b9b8c21ecd5ccc7931f45b7fbb278917b9e3f98a972a1630705b5895efe35ff4"
Oct 31 01:40:30.162018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.161963    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7" podStartSLOduration=1.161946838 podStartE2EDuration="1.161946838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:30.135838748 +0000 UTC m=+1188.084487163" watchObservedRunningTime="2026-10-31 01:40:30.161946838 +0000 UTC m=+1188.110595355"
Oct 31 01:40:30.162536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.162503    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw" podStartSLOduration=1.162492755 podStartE2EDuration="1.162492755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:30.160710809 +0000 UTC m=+1188.109359202" watchObservedRunningTime="2026-10-31 01:40:30.162492755 +0000 UTC m=+1188.111141171"
Oct 31 01:40:30.170285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.170263    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-7e5955\""
Oct 31 01:40:30.170427 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.170366    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-7e5955-dockercfg-g8f84\""
Oct 31 01:40:30.175619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.175596    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85fb67f2-c591-43be-9ca0-b4857f34f0f4" path="/var/lib/kubelet/pods/85fb67f2-c591-43be-9ca0-b4857f34f0f4/volumes"
Oct 31 01:40:30.188119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.188099    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:40:30.235628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.235589    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms\" (UID: \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:30.235794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.235644    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22ffv\" (UniqueName: \"kubernetes.io/projected/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-kube-api-access-22ffv\") pod \"payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms\" (UID: \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:30.336465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.336368    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-podinfo\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.337131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.337108    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istiod-ca-cert\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.337326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.337307    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-envoy\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.337472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.337457    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-token\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-token\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.337586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.337572    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-credential-socket\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.338101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.337752    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-kube-api-access-sk7zz\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-kube-api-access-sk7zz\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.338255 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.338237    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-certs\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.345212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.338411    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-socket\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.350644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.339833    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istiod-ca-cert" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:30.350644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.340611    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-credential-socket" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:30.350644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.340773    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-certs" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:30.350862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.341262    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-socket" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:30.350862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.349379    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-token" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:30.350862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.350725    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-data\") pod \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\" (UID: \"2e792dd8-1b35-4b8b-a4cf-7ded278a4201\") "
Oct 31 01:40:30.351070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351011    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-data" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:30.351181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351155    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-envoy" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:30.351181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351164    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms\" (UID: \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:30.351295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351208    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22ffv\" (UniqueName: \"kubernetes.io/projected/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-kube-api-access-22ffv\") pod \"payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms\" (UID: \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:30.351295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351229    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-podinfo" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:40:30.351295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351263    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-kube-api-access-sk7zz" pod "2e792dd8-1b35-4b8b-a4cf-7ded278a4201" (UID: "2e792dd8-1b35-4b8b-a4cf-7ded278a4201"). InnerVolumeSpecName "kube-api-access-sk7zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:30.351429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351337    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.351429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351358    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.351429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351372    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.351429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351384    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.351429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351398    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.351429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351411    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.351429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351424    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.351820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351437    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.351820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.351796    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms\" (UID: \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:30.364097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.364075    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22ffv\" (UniqueName: \"kubernetes.io/projected/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-kube-api-access-22ffv\") pod \"payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms\" (UID: \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:30.444961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.444930    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:30.452594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.452575    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7zz\" (UniqueName: \"kubernetes.io/projected/2e792dd8-1b35-4b8b-a4cf-7ded278a4201-kube-api-access-sk7zz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:30.756304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.756271    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"]
Oct 31 01:40:30.756762 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.756679    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy"
Oct 31 01:40:30.756762 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.756693    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy"
Oct 31 01:40:30.756762 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.756722    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy"
Oct 31 01:40:30.756762 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.756727    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy"
Oct 31 01:40:30.756972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.756790    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy"
Oct 31 01:40:30.761427 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.761009    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:30.762638 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.762615    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"]
Oct 31 01:40:30.770128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.770106    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3cf6f4\""
Oct 31 01:40:30.770511 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:30.770474    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918c7c8a_23e4_4322_862b_d4f7b5de0fc3.slice/crio-edc3c1788de7adcc256028761f19273c1cf38f27e50f9cfc737c1b05e9bc9d85 WatchSource:0}: Error finding container edc3c1788de7adcc256028761f19273c1cf38f27e50f9cfc737c1b05e9bc9d85: Status 404 returned error can't find the container with id edc3c1788de7adcc256028761f19273c1cf38f27e50f9cfc737c1b05e9bc9d85
Oct 31 01:40:30.773091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.772247    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"]
Oct 31 01:40:30.788318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.788293    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3cf6f4-dockercfg-nmgvq\""
Oct 31 01:40:30.829981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.817151    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"]
Oct 31 01:40:30.829981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.817939    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" containerName="istio-proxy"
Oct 31 01:40:30.829981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.825304    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:30.829981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.825985    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"]
Oct 31 01:40:30.857081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.857031    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdvn\" (UniqueName: \"kubernetes.io/projected/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-kube-api-access-wxdvn\") pod \"payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929\" (UID: \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:30.857202 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.857097    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwfx\" (UniqueName: \"kubernetes.io/projected/be9d1606-5c4a-4d16-8370-958320546e86-kube-api-access-kqwfx\") pod \"payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6\" (UID: \"be9d1606-5c4a-4d16-8370-958320546e86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:30.857202 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.857148    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929\" (UID: \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:30.857202 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.857196    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9d1606-5c4a-4d16-8370-958320546e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6\" (UID: \"be9d1606-5c4a-4d16-8370-958320546e86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:30.957804 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.957778    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9d1606-5c4a-4d16-8370-958320546e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6\" (UID: \"be9d1606-5c4a-4d16-8370-958320546e86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:30.957967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.957955    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdvn\" (UniqueName: \"kubernetes.io/projected/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-kube-api-access-wxdvn\") pod \"payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929\" (UID: \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:30.958111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.958097    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwfx\" (UniqueName: \"kubernetes.io/projected/be9d1606-5c4a-4d16-8370-958320546e86-kube-api-access-kqwfx\") pod \"payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6\" (UID: \"be9d1606-5c4a-4d16-8370-958320546e86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:30.958264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.958249    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929\" (UID: \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:30.958904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.958880    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9d1606-5c4a-4d16-8370-958320546e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6\" (UID: \"be9d1606-5c4a-4d16-8370-958320546e86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:30.959155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.959138    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929\" (UID: \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:30.970104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.969877    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwfx\" (UniqueName: \"kubernetes.io/projected/be9d1606-5c4a-4d16-8370-958320546e86-kube-api-access-kqwfx\") pod \"payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6\" (UID: \"be9d1606-5c4a-4d16-8370-958320546e86\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:30.970482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:30.970462    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdvn\" (UniqueName: \"kubernetes.io/projected/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-kube-api-access-wxdvn\") pod \"payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929\" (UID: \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:31.092236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.092180    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:31.125064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.124342    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms" event={"ID":"918c7c8a-23e4-4322-862b-d4f7b5de0fc3","Type":"ContainerStarted","Data":"9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25"}
Oct 31 01:40:31.125064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.124411    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms" event={"ID":"918c7c8a-23e4-4322-862b-d4f7b5de0fc3","Type":"ContainerStarted","Data":"edc3c1788de7adcc256028761f19273c1cf38f27e50f9cfc737c1b05e9bc9d85"}
Oct 31 01:40:31.129063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.126275    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:31.129063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.128022    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"
Oct 31 01:40:31.129254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.129150    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl" event={"ID":"2e792dd8-1b35-4b8b-a4cf-7ded278a4201","Type":"ContainerDied","Data":"f4a4e7ffaf5730e805cb7ba01d678547609e6c67caf9c08f3ed3068fbcb2e62c"}
Oct 31 01:40:31.129254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.129209    5293 scope.go:122] "RemoveContainer" containerID="427411b43236d79647eb9a4c19606a5da0b231fd487519be812855996548d801"
Oct 31 01:40:31.129360 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.129292    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:31.144538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.144488    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms" podStartSLOduration=1.144470257 podStartE2EDuration="1.144470257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:31.142606812 +0000 UTC m=+1189.091255211" watchObservedRunningTime="2026-10-31 01:40:31.144470257 +0000 UTC m=+1189.093118650"
Oct 31 01:40:31.155612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.155586    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"]
Oct 31 01:40:31.162913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.161812    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6198e156-openshift-default-59754d6d67-zkkdl"]
Oct 31 01:40:31.188431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.188404    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:31.427116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.415097    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"]
Oct 31 01:40:31.431158 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:31.431125    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9d1606_5c4a_4d16_8370_958320546e86.slice/crio-532647a816a88fe0cb4225a2a6511cf144e2fec567385eb1b5d07a862e04dcd3 WatchSource:0}: Error finding container 532647a816a88fe0cb4225a2a6511cf144e2fec567385eb1b5d07a862e04dcd3: Status 404 returned error can't find the container with id 532647a816a88fe0cb4225a2a6511cf144e2fec567385eb1b5d07a862e04dcd3
Oct 31 01:40:31.507065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:31.503868    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"]
Oct 31 01:40:31.531975 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:31.531935    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaae2c1d_05e9_4a5f_892a_2f454f4b1f76.slice/crio-f1a332d8a79b5d900b8962eec3a3fea7f3a79553ed0a3e9983b4af38736f1633 WatchSource:0}: Error finding container f1a332d8a79b5d900b8962eec3a3fea7f3a79553ed0a3e9983b4af38736f1633: Status 404 returned error can't find the container with id f1a332d8a79b5d900b8962eec3a3fea7f3a79553ed0a3e9983b4af38736f1633
Oct 31 01:40:32.136685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.136645    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929" event={"ID":"daae2c1d-05e9-4a5f-892a-2f454f4b1f76","Type":"ContainerStarted","Data":"ca0b1f4b59afd4f8563909be2d63776daa6c07f674ca54e579dbfc35cbef1c78"}
Oct 31 01:40:32.137189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.136839    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:32.137189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.136872    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929" event={"ID":"daae2c1d-05e9-4a5f-892a-2f454f4b1f76","Type":"ContainerStarted","Data":"f1a332d8a79b5d900b8962eec3a3fea7f3a79553ed0a3e9983b4af38736f1633"}
Oct 31 01:40:32.138462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.138428    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6" event={"ID":"be9d1606-5c4a-4d16-8370-958320546e86","Type":"ContainerStarted","Data":"a95a3da163e34fd9c9813dad9d604445d588a90d9532d56358a0286023b07be5"}
Oct 31 01:40:32.138585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.138485    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6" event={"ID":"be9d1606-5c4a-4d16-8370-958320546e86","Type":"ContainerStarted","Data":"532647a816a88fe0cb4225a2a6511cf144e2fec567385eb1b5d07a862e04dcd3"}
Oct 31 01:40:32.138585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.138513    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:32.154449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.154382    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929" podStartSLOduration=2.154365582 podStartE2EDuration="2.154365582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:32.153137063 +0000 UTC m=+1190.101785448" watchObservedRunningTime="2026-10-31 01:40:32.154365582 +0000 UTC m=+1190.103013975"
Oct 31 01:40:32.170580 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.170533    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6" podStartSLOduration=2.170516813 podStartE2EDuration="2.170516813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:32.16918104 +0000 UTC m=+1190.117829431" watchObservedRunningTime="2026-10-31 01:40:32.170516813 +0000 UTC m=+1190.119165206"
Oct 31 01:40:32.175386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:32.175365    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e792dd8-1b35-4b8b-a4cf-7ded278a4201" path="/var/lib/kubelet/pods/2e792dd8-1b35-4b8b-a4cf-7ded278a4201/volumes"
Oct 31 01:40:36.882511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:36.882475    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:40:38.052186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:38.052154    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:40:38.052570 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:38.052209    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:40:40.104142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:40.104108    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 01:40:40.104682 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:40.104648    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 01:40:40.203913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:40.203883    5293 generic.go:356] "Generic (PLEG): container finished" podID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerID="823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686" exitCode=0
Oct 31 01:40:40.204092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:40.203957    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" event={"ID":"177634d2-0f59-400b-8ad4-77188f8ce3cf","Type":"ContainerDied","Data":"823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686"}
Oct 31 01:40:40.204402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:40.204384    5293 scope.go:122] "RemoveContainer" containerID="823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686"
Oct 31 01:40:40.205897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:40.205839    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:40.206026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:40.205944    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:40.206026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:40.205995    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:41.231594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.231551    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" event={"ID":"177634d2-0f59-400b-8ad4-77188f8ce3cf","Type":"ContainerStarted","Data":"db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088"}
Oct 31 01:40:41.458934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.458900    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"]
Oct 31 01:40:41.463549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.463524    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.467535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.467513    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"default-dockercfg-gn5q2\""
Oct 31 01:40:41.467646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.467580    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"kube-root-ca.crt\""
Oct 31 01:40:41.470326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.470308    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"openshift-service-ca.crt\""
Oct 31 01:40:41.474099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.474069    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"]
Oct 31 01:40:41.506658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.506551    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"simulator-w4-be6935-kserve-self-signed-certs\""
Oct 31 01:40:41.547682 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.547646    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tls-certs\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.547841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.547710    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tmp-dir\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.547841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.547810    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-home\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.547919 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.547840    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2fkr\" (UniqueName: \"kubernetes.io/projected/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-kube-api-access-g2fkr\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.547919 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.547858    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-model-cache\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.547982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.547911    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-dshm\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.650029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.649983    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fkr\" (UniqueName: \"kubernetes.io/projected/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-kube-api-access-g2fkr\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.650634 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.650607    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-model-cache\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.650813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.650796    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-dshm\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.651025 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.651006    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tls-certs\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.651196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.651179    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tmp-dir\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.651386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.651371    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-home\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.654304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.651769    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-model-cache\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.654691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.654673    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-home\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.656640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.654953    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tmp-dir\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.665867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.665828    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-dshm\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.666158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.666139    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tls-certs\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.672568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.672204    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:41.672568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.672311    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:40:41.672568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.672407    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:40:41.673713 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.673686    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2fkr\" (UniqueName: \"kubernetes.io/projected/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-kube-api-access-g2fkr\") pod \"simulator-w4-be6935-kserve-84d7b97987-pcxvb\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") " pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.778281 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:41.777887    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:41.778281 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:40:41.777890    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:40:42.084286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.084207    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:40:42.084286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.084267    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:40:42.126941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.126914    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"]
Oct 31 01:40:42.144300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.144276    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 01:40:42.145678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.145650    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 01:40:42.282523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.282484    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" event={"ID":"6817381f-1ab2-4b8b-b580-d3b37ee07f6d","Type":"ContainerStarted","Data":"3956ee9c10bbf64a8ace50ef650d6ede898773d492cf7d739144061eb1734763"}
Oct 31 01:40:42.301541 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.301487    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podStartSLOduration=1.301468204 podStartE2EDuration="1.301468204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:42.300387496 +0000 UTC m=+1200.249035889" watchObservedRunningTime="2026-10-31 01:40:42.301468204 +0000 UTC m=+1200.250116597"
Oct 31 01:40:42.671631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.671589    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:40:42.671809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.671650    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:40:42.778127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:42.778091    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:43.151318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:43.150591    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 01:40:43.151318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:43.150933    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 01:40:43.151318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:43.151007    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 01:40:43.293018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:43.291846    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" event={"ID":"6817381f-1ab2-4b8b-b580-d3b37ee07f6d","Type":"ContainerStarted","Data":"2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a"}
Oct 31 01:40:43.672536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:43.672504    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:40:43.672697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:43.672551    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:40:44.321065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.320115    5293 generic.go:356] "Generic (PLEG): container finished" podID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerID="fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8" exitCode=0
Oct 31 01:40:44.321065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.320943    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" event={"ID":"fa358a0d-fd41-42a0-bbac-52875ea48c46","Type":"ContainerDied","Data":"fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8"}
Oct 31 01:40:44.321622 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.321321    5293 scope.go:122] "RemoveContainer" containerID="fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8"
Oct 31 01:40:44.323357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.322841    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:44.323357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.322935    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:44.323357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.322986    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:40:44.332989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.332965    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:40:44.672056 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.672015    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:40:44.672257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:44.672099    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:40:45.348072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:45.347175    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" event={"ID":"fa358a0d-fd41-42a0-bbac-52875ea48c46","Type":"ContainerStarted","Data":"e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5"}
Oct 31 01:40:45.519470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:45.519438    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:45.519819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:45.519795    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 01:40:45.519971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:45.519947    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 01:40:45.671913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:45.671879    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:40:45.672101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:45.671932    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:40:46.526064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:46.522427    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 01:40:46.526064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:46.522484    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 01:40:46.672649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:46.672610    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:40:46.672812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:46.672665    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:40:47.521417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:47.521382    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 01:40:47.521695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:47.521665    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 01:40:47.676259 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:47.676229    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:47.676649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:47.676611    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:47.677386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:47.677367    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 01:40:48.519453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:48.519417    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 01:40:48.519722 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:48.519694    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 01:40:49.521366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:49.521327    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 01:40:49.521892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:49.521379    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 01:40:50.520190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:50.520149    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 01:40:50.520377 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:50.520216    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 01:40:51.028404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.028368    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"]
Oct 31 01:40:51.033067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.033022    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.041144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.040736    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"simulator-w3-7e5955-kserve-self-signed-certs\""
Oct 31 01:40:51.041144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.041121    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"default-dockercfg-2xhns\""
Oct 31 01:40:51.041406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.041289    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"openshift-service-ca.crt\""
Oct 31 01:40:51.041467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.041438    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"kube-root-ca.crt\""
Oct 31 01:40:51.053813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.053793    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"]
Oct 31 01:40:51.088106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.088081    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-tmp-dir\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.088241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.088120    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-model-cache\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.088292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.088224    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcabdce-f8cb-420c-9f80-116695aea37e-tls-certs\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.088292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.088265    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-home\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.088292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.088283    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-dshm\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.088432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.088408    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmf7\" (UniqueName: \"kubernetes.io/projected/6dcabdce-f8cb-420c-9f80-116695aea37e-kube-api-access-ghmf7\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189090    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmf7\" (UniqueName: \"kubernetes.io/projected/6dcabdce-f8cb-420c-9f80-116695aea37e-kube-api-access-ghmf7\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189168    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-tmp-dir\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189202    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-model-cache\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189268    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcabdce-f8cb-420c-9f80-116695aea37e-tls-certs\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189293    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-home\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189311    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-dshm\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189653    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-model-cache\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189747 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189672    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-tmp-dir\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.189747 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.189697    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-home\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.191890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.191863    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-dshm\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.192132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.192118    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcabdce-f8cb-420c-9f80-116695aea37e-tls-certs\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.200423 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.200402    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmf7\" (UniqueName: \"kubernetes.io/projected/6dcabdce-f8cb-420c-9f80-116695aea37e-kube-api-access-ghmf7\") pod \"simulator-w3-7e5955-kserve-67fcf8454b-72v9n\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") " pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.346394 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.346316    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:40:51.377369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.377338    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:40:51.524011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.523979    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:51.524451 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.524429    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:51.525143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.525127    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 01:40:51.635578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.635464    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"]
Oct 31 01:40:51.638429 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:40:51.638396    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dcabdce_f8cb_420c_9f80_116695aea37e.slice/crio-082c9801e043ef6fb9fc4997709182cb18bc5536e957e536f3d593c6323f2030 WatchSource:0}: Error finding container 082c9801e043ef6fb9fc4997709182cb18bc5536e957e536f3d593c6323f2030: Status 404 returned error can't find the container with id 082c9801e043ef6fb9fc4997709182cb18bc5536e957e536f3d593c6323f2030
Oct 31 01:40:51.778113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.778082    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:51.793302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:51.793271    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:52.410175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:52.410132    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" event={"ID":"6dcabdce-f8cb-420c-9f80-116695aea37e","Type":"ContainerStarted","Data":"298c67d5193793144a5c46685c27b1286f6dbfe524b0f40ba5e5251c3fd411f4"}
Oct 31 01:40:52.410656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:52.410184    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" event={"ID":"6dcabdce-f8cb-420c-9f80-116695aea37e","Type":"ContainerStarted","Data":"082c9801e043ef6fb9fc4997709182cb18bc5536e957e536f3d593c6323f2030"}
Oct 31 01:40:52.421780 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:52.421757    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:40:52.428651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:52.428585    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podStartSLOduration=1.428572216 podStartE2EDuration="1.428572216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:52.428233115 +0000 UTC m=+1210.376881508" watchObservedRunningTime="2026-10-31 01:40:52.428572216 +0000 UTC m=+1210.377220608"
Oct 31 01:40:53.359082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:40:53.356269    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:41:00.700565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:00.700525    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:41:01.346988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:01.346957    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:41:01.359957 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:01.359930    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:41:01.480338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:01.480309    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:41:02.455807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.455777    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"]
Oct 31 01:41:02.459913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.459890    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.478457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.478433    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"premium-w4-be6935-kserve-self-signed-certs\""
Oct 31 01:41:02.506695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.506665    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"]
Oct 31 01:41:02.628359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.628326    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-tmp-dir\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.628533 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.628453    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-dshm\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.628592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.628575    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-model-cache\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.628630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.628616    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxwd\" (UniqueName: \"kubernetes.io/projected/b4cc03be-968e-4464-9710-4d082d728072-kube-api-access-fpxwd\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.628732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.628713    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-home\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.628782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.628745    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cc03be-968e-4464-9710-4d082d728072-tls-certs\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.729824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.729722    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-dshm\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.730014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.729824    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-model-cache\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.730014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.729865    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxwd\" (UniqueName: \"kubernetes.io/projected/b4cc03be-968e-4464-9710-4d082d728072-kube-api-access-fpxwd\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.730014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.729901    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-home\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.730014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.729927    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cc03be-968e-4464-9710-4d082d728072-tls-certs\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.730014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.729965    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-tmp-dir\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.730321 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.730302    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-model-cache\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.730389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.730366    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-home\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.730453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.730427    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-tmp-dir\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.732553 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.732532    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-dshm\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.733026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.733003    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cc03be-968e-4464-9710-4d082d728072-tls-certs\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.741519 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.741496    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxwd\" (UniqueName: \"kubernetes.io/projected/b4cc03be-968e-4464-9710-4d082d728072-kube-api-access-fpxwd\") pod \"premium-w4-be6935-kserve-7766745998-74lsw\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") " pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:02.773163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:02.773143    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:03.058712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:03.058611    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"]
Oct 31 01:41:03.061342 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:41:03.061317    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4cc03be_968e_4464_9710_4d082d728072.slice/crio-d01c2bc0fe8ead32385a7ff84f58818d961bed12ecaabb8d12f270c03e2f60fd WatchSource:0}: Error finding container d01c2bc0fe8ead32385a7ff84f58818d961bed12ecaabb8d12f270c03e2f60fd: Status 404 returned error can't find the container with id d01c2bc0fe8ead32385a7ff84f58818d961bed12ecaabb8d12f270c03e2f60fd
Oct 31 01:41:03.478081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:03.478020    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" event={"ID":"b4cc03be-968e-4464-9710-4d082d728072","Type":"ContainerStarted","Data":"60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a"}
Oct 31 01:41:03.478479 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:03.478097    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" event={"ID":"b4cc03be-968e-4464-9710-4d082d728072","Type":"ContainerStarted","Data":"d01c2bc0fe8ead32385a7ff84f58818d961bed12ecaabb8d12f270c03e2f60fd"}
Oct 31 01:41:03.497925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:03.497871    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podStartSLOduration=1.497848227 podStartE2EDuration="1.497848227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:03.49676056 +0000 UTC m=+1221.445408953" watchObservedRunningTime="2026-10-31 01:41:03.497848227 +0000 UTC m=+1221.446496629"
Oct 31 01:41:03.773980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:03.773911    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:08.910272 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:08.910235    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"]
Oct 31 01:41:08.913999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:08.913983    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:08.919388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:08.919363    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"default-dockercfg-rspfz\""
Oct 31 01:41:08.958843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:08.958819    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"kube-root-ca.crt\""
Oct 31 01:41:08.968279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:08.968252    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"openshift-service-ca.crt\""
Oct 31 01:41:08.972818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:08.972796    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"]
Oct 31 01:41:09.009411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.009380    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-model-cache\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.009551 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.009432    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ef0574-059a-42a7-b771-a474c4ca214b-tls-certs\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.009644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.009627    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-dshm\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.009705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.009678    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-tmp-dir\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.009764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.009744    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-home\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.009817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.009771    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7ms\" (UniqueName: \"kubernetes.io/projected/d8ef0574-059a-42a7-b771-a474c4ca214b-kube-api-access-5s7ms\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.017906 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.017883    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"premium-w1-0e8b5c-kserve-self-signed-certs\""
Oct 31 01:41:09.111088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111029    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-model-cache\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.111271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111121    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ef0574-059a-42a7-b771-a474c4ca214b-tls-certs\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.111271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111193    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-dshm\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.111271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111218    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-tmp-dir\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.111271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111236    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-home\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.111271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111250    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7ms\" (UniqueName: \"kubernetes.io/projected/d8ef0574-059a-42a7-b771-a474c4ca214b-kube-api-access-5s7ms\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.111547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111524    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-model-cache\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.111872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111846    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-tmp-dir\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.111872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.111859    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-home\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.113759 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.113729    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-dshm\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.113892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.113879    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ef0574-059a-42a7-b771-a474c4ca214b-tls-certs\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.125650 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.125622    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7ms\" (UniqueName: \"kubernetes.io/projected/d8ef0574-059a-42a7-b771-a474c4ca214b-kube-api-access-5s7ms\") pod \"premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") " pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.226260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.226183    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:09.511332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:09.511244    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"]
Oct 31 01:41:09.514967 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:41:09.514937    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8ef0574_059a_42a7_b771_a474c4ca214b.slice/crio-83347d21efa4cfd2a268e8abe57f5e351c24b53d21cb18f9f25b22ab344344c0 WatchSource:0}: Error finding container 83347d21efa4cfd2a268e8abe57f5e351c24b53d21cb18f9f25b22ab344344c0: Status 404 returned error can't find the container with id 83347d21efa4cfd2a268e8abe57f5e351c24b53d21cb18f9f25b22ab344344c0
Oct 31 01:41:10.525516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:10.525475    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" event={"ID":"d8ef0574-059a-42a7-b771-a474c4ca214b","Type":"ContainerStarted","Data":"2922014f26b988552436afdce7ef37a5606c2cf1d9828c4eba7a0cff655ff032"}
Oct 31 01:41:10.525953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:10.525538    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" event={"ID":"d8ef0574-059a-42a7-b771-a474c4ca214b","Type":"ContainerStarted","Data":"83347d21efa4cfd2a268e8abe57f5e351c24b53d21cb18f9f25b22ab344344c0"}
Oct 31 01:41:10.544959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:10.544911    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podStartSLOduration=2.5448927550000002 podStartE2EDuration="2.544892755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:10.542264003 +0000 UTC m=+1228.490912395" watchObservedRunningTime="2026-10-31 01:41:10.544892755 +0000 UTC m=+1228.493541148"
Oct 31 01:41:10.713563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:10.713530    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:41:11.226687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:11.226655    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:11.777808 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:41:11.777775    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:41:12.774222 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:12.774184    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:12.787261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:12.787226    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:13.563986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:13.563957    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:41:16.709338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.709311    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"]
Oct 31 01:41:16.714175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.714155    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.717163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.717139    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"openshift-service-ca.crt\""
Oct 31 01:41:16.717318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.717295    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"kube-root-ca.crt\""
Oct 31 01:41:16.717475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.717447    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"premium-w0-3cf6f4-kserve-self-signed-certs\""
Oct 31 01:41:16.717475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.717453    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"default-dockercfg-z5c5f\""
Oct 31 01:41:16.728143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.724120    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"]
Oct 31 01:41:16.815723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.815690    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-model-cache\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.815723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.815729    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-home\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.815946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.815755    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-dshm\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.815946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.815903    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7e17ef2-7de5-45d9-85cd-a339307d8477-tls-certs\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.815946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.815930    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-tmp-dir\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.816108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.816065    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486bj\" (UniqueName: \"kubernetes.io/projected/c7e17ef2-7de5-45d9-85cd-a339307d8477-kube-api-access-486bj\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.916761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.916721    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7e17ef2-7de5-45d9-85cd-a339307d8477-tls-certs\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.916977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.916775    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-tmp-dir\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.916977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.916858    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486bj\" (UniqueName: \"kubernetes.io/projected/c7e17ef2-7de5-45d9-85cd-a339307d8477-kube-api-access-486bj\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.916977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.916928    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-model-cache\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.916977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.916964    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-home\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.917245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.917008    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-dshm\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.917425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.917371    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-model-cache\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.917425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.917382    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-home\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.917560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.917473    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-tmp-dir\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.919416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.919392    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-dshm\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.919416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.919408    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7e17ef2-7de5-45d9-85cd-a339307d8477-tls-certs\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:16.928486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:16.928463    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-486bj\" (UniqueName: \"kubernetes.io/projected/c7e17ef2-7de5-45d9-85cd-a339307d8477-kube-api-access-486bj\") pod \"premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") " pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:17.031089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:17.030993    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:17.310380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:17.310352    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"]
Oct 31 01:41:17.312620 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:41:17.312595    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7e17ef2_7de5_45d9_85cd_a339307d8477.slice/crio-46db11f7c24d26d1be207bc79b11708ca2a3e62983c230f561e883ff89965d3e WatchSource:0}: Error finding container 46db11f7c24d26d1be207bc79b11708ca2a3e62983c230f561e883ff89965d3e: Status 404 returned error can't find the container with id 46db11f7c24d26d1be207bc79b11708ca2a3e62983c230f561e883ff89965d3e
Oct 31 01:41:17.597107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:17.596996    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" event={"ID":"c7e17ef2-7de5-45d9-85cd-a339307d8477","Type":"ContainerStarted","Data":"834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132"}
Oct 31 01:41:17.597107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:17.597067    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" event={"ID":"c7e17ef2-7de5-45d9-85cd-a339307d8477","Type":"ContainerStarted","Data":"46db11f7c24d26d1be207bc79b11708ca2a3e62983c230f561e883ff89965d3e"}
Oct 31 01:41:17.619417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:17.619364    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podStartSLOduration=1.619346584 podStartE2EDuration="1.619346584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:17.618275192 +0000 UTC m=+1235.566923589" watchObservedRunningTime="2026-10-31 01:41:17.619346584 +0000 UTC m=+1235.567994977"
Oct 31 01:41:18.032093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:18.032057    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:19.226859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:19.226824    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:19.239943 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:19.239915    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:19.625680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:19.625603    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:41:20.270266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:20.270238    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:41:27.031481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:27.031441    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:27.049221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:27.049189    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:27.669700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:27.669672    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:41:35.106345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:35.106315    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:41:39.465655 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.465614    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"]
Oct 31 01:41:39.469990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.469962    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.473192 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.473171    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7e5955-openshift-default-dockercfg-t99cw\""
Oct 31 01:41:39.483184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.483160    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"]
Oct 31 01:41:39.580220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580160    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.580413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580278    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.580413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580353    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.580413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580387    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293991cd-eab8-48f3-882f-bf1dc275809d-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.580583 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580443    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.580633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580574    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.580686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580632    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.580686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580656    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293991cd-eab8-48f3-882f-bf1dc275809d-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.580792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.580688    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jqp\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-kube-api-access-j5jqp\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.681538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681504    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293991cd-eab8-48f3-882f-bf1dc275809d-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.681538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681539    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.681733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681591    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.681776 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681748    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.681811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681783    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293991cd-eab8-48f3-882f-bf1dc275809d-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.681811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681802    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jqp\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-kube-api-access-j5jqp\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.681899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681847    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.681899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681883    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.682002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.681971    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.682179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.682148    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.682276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.682255    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.682562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.682542    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293991cd-eab8-48f3-882f-bf1dc275809d-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.682610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.682581    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.682708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.682688    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.684136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.684111    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.684343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.684327    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293991cd-eab8-48f3-882f-bf1dc275809d-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.693055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.693020    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.693175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.693158    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jqp\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-kube-api-access-j5jqp\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:39.785851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:39.785772    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:40.087651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:40.087621    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"]
Oct 31 01:41:40.089635 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:41:40.089603    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293991cd_eab8_48f3_882f_bf1dc275809d.slice/crio-0df7871af3eef04f8d41d962f9cc610fd887907511a6437425bef9f9d1774bd9 WatchSource:0}: Error finding container 0df7871af3eef04f8d41d962f9cc610fd887907511a6437425bef9f9d1774bd9: Status 404 returned error can't find the container with id 0df7871af3eef04f8d41d962f9cc610fd887907511a6437425bef9f9d1774bd9
Oct 31 01:41:40.092110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:40.092068    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:40.092213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:40.092190    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:40.092265 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:40.092248    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:40.734110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:40.734074    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" event={"ID":"293991cd-eab8-48f3-882f-bf1dc275809d","Type":"ContainerStarted","Data":"1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68"}
Oct 31 01:41:40.734636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:40.734130    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" event={"ID":"293991cd-eab8-48f3-882f-bf1dc275809d","Type":"ContainerStarted","Data":"0df7871af3eef04f8d41d962f9cc610fd887907511a6437425bef9f9d1774bd9"}
Oct 31 01:41:40.756966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:40.756909    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" podStartSLOduration=1.756894084 podStartE2EDuration="1.756894084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:40.755926147 +0000 UTC m=+1258.704574538" watchObservedRunningTime="2026-10-31 01:41:40.756894084 +0000 UTC m=+1258.705542546"
Oct 31 01:41:40.786512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:40.786486    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:41.777467 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:41:41.777430    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:41:41.788067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:41.788015    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:41:41.788221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:41.788097    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 01:41:42.786755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:42.786716    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:41:42.787152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:42.786813    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 01:41:43.786228 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:43.786194    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:41:43.786513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:43.786260    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 01:41:44.786350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.786317    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:41:44.786733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.786365    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 01:41:44.953930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.953899    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"]
Oct 31 01:41:44.959014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.958991    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:44.965373 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.965311    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:41:44.965524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.965413    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:41:44.965524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.965522    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:41:44.965647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.965593    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:41:44.967940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:44.967917    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"]
Oct 31 01:41:45.079766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.079685    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.079910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.079880    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg82v\" (UniqueName: \"kubernetes.io/projected/6ca43a12-5791-41a3-8aea-6602f8ef4aef-kube-api-access-xg82v\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.080031 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.080015    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.181274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.181241    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg82v\" (UniqueName: \"kubernetes.io/projected/6ca43a12-5791-41a3-8aea-6602f8ef4aef-kube-api-access-xg82v\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.181428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.181336    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.181428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.181382    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.182133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.182095    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.183896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.183874    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.194290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.194272    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg82v\" (UniqueName: \"kubernetes.io/projected/6ca43a12-5791-41a3-8aea-6602f8ef4aef-kube-api-access-xg82v\") pod \"maas-api-revoke-sub-b755ceec-j2dbq\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.281137 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.281103    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:45.605128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.598499    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"]
Oct 31 01:41:45.630213 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:41:45.630181    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca43a12_5791_41a3_8aea_6602f8ef4aef.slice/crio-8ffd92f70f866f3fe6e60217d7a9befe0b2769ccfe19e158ae1d353a8e96cf34 WatchSource:0}: Error finding container 8ffd92f70f866f3fe6e60217d7a9befe0b2769ccfe19e158ae1d353a8e96cf34: Status 404 returned error can't find the container with id 8ffd92f70f866f3fe6e60217d7a9befe0b2769ccfe19e158ae1d353a8e96cf34
Oct 31 01:41:45.764399 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.764367    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq" event={"ID":"6ca43a12-5791-41a3-8aea-6602f8ef4aef","Type":"ContainerStarted","Data":"8ffd92f70f866f3fe6e60217d7a9befe0b2769ccfe19e158ae1d353a8e96cf34"}
Oct 31 01:41:45.786750 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.786718    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:41:45.787142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:45.786785    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 01:41:46.772914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:46.772884    5293 generic.go:356] "Generic (PLEG): container finished" podID="6ca43a12-5791-41a3-8aea-6602f8ef4aef" containerID="39624fda6f7c9418bc9741cf7abee4f44f11bc71f8b218e74b4d949dee1b37bd" exitCode=0
Oct 31 01:41:46.773076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:46.772964    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq" event={"ID":"6ca43a12-5791-41a3-8aea-6602f8ef4aef","Type":"ContainerDied","Data":"39624fda6f7c9418bc9741cf7abee4f44f11bc71f8b218e74b4d949dee1b37bd"}
Oct 31 01:41:46.875959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:46.875932    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:46.876500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:46.876480    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:46.877260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:46.877239    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 01:41:47.923970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:47.923941    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:48.025592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.025551    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca43a12-5791-41a3-8aea-6602f8ef4aef-kube-api-access-xg82v\" (UniqueName: \"kubernetes.io/projected/6ca43a12-5791-41a3-8aea-6602f8ef4aef-kube-api-access-xg82v\") pod \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") "
Oct 31 01:41:48.025788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.025616    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-openshift-ca\") pod \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") "
Oct 31 01:41:48.025788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.025743    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-xks-ca\") pod \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\" (UID: \"6ca43a12-5791-41a3-8aea-6602f8ef4aef\") "
Oct 31 01:41:48.026086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.026054    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-openshift-ca" pod "6ca43a12-5791-41a3-8aea-6602f8ef4aef" (UID: "6ca43a12-5791-41a3-8aea-6602f8ef4aef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:48.028034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.028007    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-xks-ca" pod "6ca43a12-5791-41a3-8aea-6602f8ef4aef" (UID: "6ca43a12-5791-41a3-8aea-6602f8ef4aef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:41:48.028161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.028031    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca43a12-5791-41a3-8aea-6602f8ef4aef-kube-api-access-xg82v" pod "6ca43a12-5791-41a3-8aea-6602f8ef4aef" (UID: "6ca43a12-5791-41a3-8aea-6602f8ef4aef"). InnerVolumeSpecName "kube-api-access-xg82v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:48.127419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.127381    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:41:48.127419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.127412    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg82v\" (UniqueName: \"kubernetes.io/projected/6ca43a12-5791-41a3-8aea-6602f8ef4aef-kube-api-access-xg82v\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:41:48.127419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.127424    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca43a12-5791-41a3-8aea-6602f8ef4aef-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:41:48.784668 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.784640    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"
Oct 31 01:41:48.784839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.784686    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq" event={"ID":"6ca43a12-5791-41a3-8aea-6602f8ef4aef","Type":"ContainerDied","Data":"8ffd92f70f866f3fe6e60217d7a9befe0b2769ccfe19e158ae1d353a8e96cf34"}
Oct 31 01:41:48.784839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:48.784730    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ffd92f70f866f3fe6e60217d7a9befe0b2769ccfe19e158ae1d353a8e96cf34"
Oct 31 01:41:49.160848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:49.160818    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:41:52.576644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.576567    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"]
Oct 31 01:41:52.577104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.577086    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6ca43a12-5791-41a3-8aea-6602f8ef4aef" containerName="revoke-keys"
Oct 31 01:41:52.577187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.577107    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca43a12-5791-41a3-8aea-6602f8ef4aef" containerName="revoke-keys"
Oct 31 01:41:52.577237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.577193    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca43a12-5791-41a3-8aea-6602f8ef4aef" containerName="revoke-keys"
Oct 31 01:41:52.581642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.581621    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.592587 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.592566    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:41:52.593204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.593137    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:41:52.596467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.593286    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:41:52.596467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.593299    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:41:52.596751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.596733    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"]
Oct 31 01:41:52.695432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.695403    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.695432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.695434    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66vls\" (UniqueName: \"kubernetes.io/projected/ca69856c-8787-435a-9b6d-869816da8ffc-kube-api-access-66vls\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.695737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.695473    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.797300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.797258    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.797502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.797308    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66vls\" (UniqueName: \"kubernetes.io/projected/ca69856c-8787-435a-9b6d-869816da8ffc-kube-api-access-66vls\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.797502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.797372    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.798299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.798274    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.799921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.799891    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.810045 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.810018    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66vls\" (UniqueName: \"kubernetes.io/projected/ca69856c-8787-435a-9b6d-869816da8ffc-kube-api-access-66vls\") pod \"maas-api-revoke-sub-3baee4e4-l6z7j\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:52.897442 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:52.897415    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:53.003981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.003952    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"]
Oct 31 01:41:53.009455 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.009429    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.014766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.014696    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6ae519-openshift-default-dockercfg-kmlj9\""
Oct 31 01:41:53.026250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.026217    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"]
Oct 31 01:41:53.201950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.201833    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.201932    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.202000    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.202031    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.202116    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.202143    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.202236    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.202352    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.202389    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhrs\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-kube-api-access-vjhrs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.202658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.202638    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"]
Oct 31 01:41:53.206257 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:41:53.206228    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca69856c_8787_435a_9b6d_869816da8ffc.slice/crio-3fa1c5f501e821c6f4a5f3fe8aa04f01f5a13295208334969a7a0f2b240cc0db WatchSource:0}: Error finding container 3fa1c5f501e821c6f4a5f3fe8aa04f01f5a13295208334969a7a0f2b240cc0db: Status 404 returned error can't find the container with id 3fa1c5f501e821c6f4a5f3fe8aa04f01f5a13295208334969a7a0f2b240cc0db
Oct 31 01:41:53.303468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.303436    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.303637 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.303476    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.303637 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.303519    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.303637 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.303554    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.303637 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.303576    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.303936    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.303969    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.303980    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.304065    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.304095    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhrs\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-kube-api-access-vjhrs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.304164    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.304270    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.304286    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.304803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.304606    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.306383 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.306360    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.306383 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.306376    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.315475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.315454    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.315656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.315638    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhrs\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-kube-api-access-vjhrs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.329570 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.329544    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:53.621272 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:41:53.621241    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d5a1ff_ad96_43bd_b3ee_b33072a1cb55.slice/crio-d58a07ddfebd90956e79563a9d6daac610f1b10f83ac7e112ac22d01a7251772 WatchSource:0}: Error finding container d58a07ddfebd90956e79563a9d6daac610f1b10f83ac7e112ac22d01a7251772: Status 404 returned error can't find the container with id d58a07ddfebd90956e79563a9d6daac610f1b10f83ac7e112ac22d01a7251772
Oct 31 01:41:53.621686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.621287    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"]
Oct 31 01:41:53.623815 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.623780    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:53.623941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.623840    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:53.623941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.623870    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:53.819250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.818242    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" event={"ID":"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55","Type":"ContainerStarted","Data":"f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29"}
Oct 31 01:41:53.819250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.818309    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" event={"ID":"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55","Type":"ContainerStarted","Data":"d58a07ddfebd90956e79563a9d6daac610f1b10f83ac7e112ac22d01a7251772"}
Oct 31 01:41:53.820862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.820838    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j" event={"ID":"ca69856c-8787-435a-9b6d-869816da8ffc","Type":"ContainerStarted","Data":"acc6718d5b7a087fd7d6f7c6e46425f7a9865e49464428a2dceb586c1f7ed95c"}
Oct 31 01:41:53.821010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.820999    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j" event={"ID":"ca69856c-8787-435a-9b6d-869816da8ffc","Type":"ContainerStarted","Data":"3fa1c5f501e821c6f4a5f3fe8aa04f01f5a13295208334969a7a0f2b240cc0db"}
Oct 31 01:41:53.842618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.842562    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podStartSLOduration=1.8425508320000001 podStartE2EDuration="1.842550832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:53.840101496 +0000 UTC m=+1271.788750085" watchObservedRunningTime="2026-10-31 01:41:53.842550832 +0000 UTC m=+1271.791199224"
Oct 31 01:41:53.859572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:53.859530    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j" podStartSLOduration=1.859516978 podStartE2EDuration="1.859516978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:53.857604231 +0000 UTC m=+1271.806252624" watchObservedRunningTime="2026-10-31 01:41:53.859516978 +0000 UTC m=+1271.808165402"
Oct 31 01:41:54.331065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:54.330639    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:41:54.333061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:54.332619    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:41:54.333061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:54.332696    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 01:41:54.830613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:54.830583    5293 generic.go:356] "Generic (PLEG): container finished" podID="ca69856c-8787-435a-9b6d-869816da8ffc" containerID="acc6718d5b7a087fd7d6f7c6e46425f7a9865e49464428a2dceb586c1f7ed95c" exitCode=0
Oct 31 01:41:54.831159 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:54.830673    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j" event={"ID":"ca69856c-8787-435a-9b6d-869816da8ffc","Type":"ContainerDied","Data":"acc6718d5b7a087fd7d6f7c6e46425f7a9865e49464428a2dceb586c1f7ed95c"}
Oct 31 01:41:55.329983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:55.329950    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:41:55.330231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:55.330006    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 01:41:56.076839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.076809    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:56.172139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.172108    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-openshift-ca\") pod \"ca69856c-8787-435a-9b6d-869816da8ffc\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") "
Oct 31 01:41:56.172324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.172152    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-xks-ca\") pod \"ca69856c-8787-435a-9b6d-869816da8ffc\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") "
Oct 31 01:41:56.172324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.172177    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca69856c-8787-435a-9b6d-869816da8ffc-kube-api-access-66vls\" (UniqueName: \"kubernetes.io/projected/ca69856c-8787-435a-9b6d-869816da8ffc-kube-api-access-66vls\") pod \"ca69856c-8787-435a-9b6d-869816da8ffc\" (UID: \"ca69856c-8787-435a-9b6d-869816da8ffc\") "
Oct 31 01:41:56.172462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.172441    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-openshift-ca" pod "ca69856c-8787-435a-9b6d-869816da8ffc" (UID: "ca69856c-8787-435a-9b6d-869816da8ffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:56.174505 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.174478    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-xks-ca" pod "ca69856c-8787-435a-9b6d-869816da8ffc" (UID: "ca69856c-8787-435a-9b6d-869816da8ffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:41:56.174505 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.174499    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca69856c-8787-435a-9b6d-869816da8ffc-kube-api-access-66vls" pod "ca69856c-8787-435a-9b6d-869816da8ffc" (UID: "ca69856c-8787-435a-9b6d-869816da8ffc"). InnerVolumeSpecName "kube-api-access-66vls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:56.273262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.273228    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:41:56.273262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.273256    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca69856c-8787-435a-9b6d-869816da8ffc-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:41:56.273262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.273265    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66vls\" (UniqueName: \"kubernetes.io/projected/ca69856c-8787-435a-9b6d-869816da8ffc-kube-api-access-66vls\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:41:56.304084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.303427    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"]
Oct 31 01:41:56.309075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.304425    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ca69856c-8787-435a-9b6d-869816da8ffc" containerName="revoke-keys"
Oct 31 01:41:56.309075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.304458    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca69856c-8787-435a-9b6d-869816da8ffc" containerName="revoke-keys"
Oct 31 01:41:56.309075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.304620    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca69856c-8787-435a-9b6d-869816da8ffc" containerName="revoke-keys"
Oct 31 01:41:56.318408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.318390    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.322395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.322324    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"]
Oct 31 01:41:56.337665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.337411    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:41:56.337665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.337460    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 01:41:56.337665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.337553    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3cf6f4-openshift-default-dockercfg-5t5rj\""
Oct 31 01:41:56.374034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374007    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.374194 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374074    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.374194 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374141    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.374194 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374179    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.374324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374202    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.374324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374264    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29q5g\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-kube-api-access-29q5g\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.374417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374401    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.374506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374458    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.374555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.374517    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475079    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475142    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475180    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475228    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475248    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475266    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29q5g\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-kube-api-access-29q5g\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475288    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475330    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475399    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475533    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475690 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475674    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.475886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.475864    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.476166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.476143    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.476824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.476746    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.478133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.478110    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.478627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.478603    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.486983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.486955    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.487273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.487251    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29q5g\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-kube-api-access-29q5g\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.651293 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.651251    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:56.859798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.859759    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j" event={"ID":"ca69856c-8787-435a-9b6d-869816da8ffc","Type":"ContainerDied","Data":"3fa1c5f501e821c6f4a5f3fe8aa04f01f5a13295208334969a7a0f2b240cc0db"}
Oct 31 01:41:56.859959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.859847    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fa1c5f501e821c6f4a5f3fe8aa04f01f5a13295208334969a7a0f2b240cc0db"
Oct 31 01:41:56.859959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.859781    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"
Oct 31 01:41:56.958221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.957940    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"]
Oct 31 01:41:56.961408 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:41:56.961375    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ba9c46_10ea_43e1_9cbb_fee917d0b0cd.slice/crio-2b9e2e2029718a071c85b6d6d1049eca5fe6747d589cc7aa3aecfdf07d1da04d WatchSource:0}: Error finding container 2b9e2e2029718a071c85b6d6d1049eca5fe6747d589cc7aa3aecfdf07d1da04d: Status 404 returned error can't find the container with id 2b9e2e2029718a071c85b6d6d1049eca5fe6747d589cc7aa3aecfdf07d1da04d
Oct 31 01:41:56.963754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.963722    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:56.963827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.963794    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:56.963864 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:56.963843    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:41:57.334113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:57.333434    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:41:57.334113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:57.333497    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 01:41:57.867295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:57.867251    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" event={"ID":"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd","Type":"ContainerStarted","Data":"a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a"}
Oct 31 01:41:57.867472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:57.867323    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" event={"ID":"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd","Type":"ContainerStarted","Data":"2b9e2e2029718a071c85b6d6d1049eca5fe6747d589cc7aa3aecfdf07d1da04d"}
Oct 31 01:41:57.890125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:57.890085    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podStartSLOduration=1.890070833 podStartE2EDuration="1.890070833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:57.888580241 +0000 UTC m=+1275.837228647" watchObservedRunningTime="2026-10-31 01:41:57.890070833 +0000 UTC m=+1275.838719204"
Oct 31 01:41:58.245063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.244803    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"]
Oct 31 01:41:58.253783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.253742    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.257441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.257072    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:41:58.258081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.257648    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:41:58.258961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.258233    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:41:58.258961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.258284    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:41:58.258961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.258837    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"]
Oct 31 01:41:58.294346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.294185    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.294346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.294240    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.294346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.294300    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwspm\" (UniqueName: \"kubernetes.io/projected/1c94eb49-dffe-47d8-8777-f1331e88b016-kube-api-access-xwspm\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.364544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.364473    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:41:58.364544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.364517    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 01:41:58.395702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.395667    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.395702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.395707    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.395951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.395727    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwspm\" (UniqueName: \"kubernetes.io/projected/1c94eb49-dffe-47d8-8777-f1331e88b016-kube-api-access-xwspm\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.405075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.396514    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.460067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.459551    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.464061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.461527    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwspm\" (UniqueName: \"kubernetes.io/projected/1c94eb49-dffe-47d8-8777-f1331e88b016-kube-api-access-xwspm\") pod \"maas-api-revoke-sub-94b8328a-ddpzf\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.570150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.570058    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:41:58.659064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.657406    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:41:58.670067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.662808    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:41:58.670067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.662865    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:41:58.991124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:58.991081    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"]
Oct 31 01:41:59.333494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:59.333403    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:41:59.333811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:59.333781    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 01:41:59.653062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:59.651870    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:41:59.653062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:59.651934    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:41:59.882073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:59.882015    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf" event={"ID":"1c94eb49-dffe-47d8-8777-f1331e88b016","Type":"ContainerStarted","Data":"eae40727a8a56fc7a7548f836baa74a42b68b390bc3c7a915e898f50a006dd58"}
Oct 31 01:41:59.882242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:41:59.882098    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf" event={"ID":"1c94eb49-dffe-47d8-8777-f1331e88b016","Type":"ContainerStarted","Data":"f111c6c60b9654ae8ce625da55de5e646ab1d7824a8a1345916f67256ca81f40"}
Oct 31 01:42:00.090109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.089980    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf" podStartSLOduration=2.089959257 podStartE2EDuration="2.089959257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:59.9040315 +0000 UTC m=+1277.852679904" watchObservedRunningTime="2026-10-31 01:42:00.089959257 +0000 UTC m=+1278.038607649"
Oct 31 01:42:00.092885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.092860    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"]
Oct 31 01:42:00.114494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.111274    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.114494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.112994    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"]
Oct 31 01:42:00.187064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.178288    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclgv\" (UniqueName: \"kubernetes.io/projected/0f6be199-c533-4e82-b569-59bd860268b8-kube-api-access-jclgv\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.187064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.178352    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.187064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.178413    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.281758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.281727    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclgv\" (UniqueName: \"kubernetes.io/projected/0f6be199-c533-4e82-b569-59bd860268b8-kube-api-access-jclgv\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.281933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.281768    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.281933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.281810    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.282522 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.282501    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.285456 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.285428    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.293825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.293761    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclgv\" (UniqueName: \"kubernetes.io/projected/0f6be199-c533-4e82-b569-59bd860268b8-kube-api-access-jclgv\") pod \"maas-api-revoke-sub-e5259240-5chbr\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.332015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.331498    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:42:00.332015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.331571    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 01:42:00.556956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.556923    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:00.563598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.563512    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"]
Oct 31 01:42:00.568723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.568697    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.581155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.581129    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"]
Oct 31 01:42:00.585589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.585555    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.585739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.585721    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrb46\" (UniqueName: \"kubernetes.io/projected/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-kube-api-access-lrb46\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.585932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.585914    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.652507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.651977    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:42:00.652507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.652028    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:42:00.687206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.687176    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.687670 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.687650    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.687825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.687806    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.687917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.687902    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrb46\" (UniqueName: \"kubernetes.io/projected/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-kube-api-access-lrb46\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.691606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.691563    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.734771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.734741    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrb46\" (UniqueName: \"kubernetes.io/projected/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-kube-api-access-lrb46\") pod \"maas-api-revoke-sub-67f2f42e-6dgtt\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:00.889807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.889730    5293 generic.go:356] "Generic (PLEG): container finished" podID="1c94eb49-dffe-47d8-8777-f1331e88b016" containerID="eae40727a8a56fc7a7548f836baa74a42b68b390bc3c7a915e898f50a006dd58" exitCode=0
Oct 31 01:42:00.889807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.889768    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf" event={"ID":"1c94eb49-dffe-47d8-8777-f1331e88b016","Type":"ContainerDied","Data":"eae40727a8a56fc7a7548f836baa74a42b68b390bc3c7a915e898f50a006dd58"}
Oct 31 01:42:00.893307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.893288    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"]
Oct 31 01:42:00.895943 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:00.895916    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6be199_c533_4e82_b569_59bd860268b8.slice/crio-aa9821ed58ed98f3a1bc89009e4000354a4959df431422ac8a0f110ee1e48035 WatchSource:0}: Error finding container aa9821ed58ed98f3a1bc89009e4000354a4959df431422ac8a0f110ee1e48035: Status 404 returned error can't find the container with id aa9821ed58ed98f3a1bc89009e4000354a4959df431422ac8a0f110ee1e48035
Oct 31 01:42:00.970086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:00.969935    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:01.284800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.284775    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"]
Oct 31 01:42:01.286876 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:01.286848    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80f96a4_dabc_4ac6_a225_76e1f3f7f860.slice/crio-247dd803acdeb5c369b678e07c0bdb1ba2f8d897d7034f1c5540f3f7aff29479 WatchSource:0}: Error finding container 247dd803acdeb5c369b678e07c0bdb1ba2f8d897d7034f1c5540f3f7aff29479: Status 404 returned error can't find the container with id 247dd803acdeb5c369b678e07c0bdb1ba2f8d897d7034f1c5540f3f7aff29479
Oct 31 01:42:01.335135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.335116    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:42:01.335489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.335464    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:42:01.336147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.336130    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:42:01.663086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.656265    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:42:01.663086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.656327    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:42:01.916297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.915904    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt" event={"ID":"d80f96a4-dabc-4ac6-a225-76e1f3f7f860","Type":"ContainerStarted","Data":"1403b5aa7dc6f16bdb5a60095785923c65a0533504baca896ea8fea92f4f9575"}
Oct 31 01:42:01.916297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.915967    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt" event={"ID":"d80f96a4-dabc-4ac6-a225-76e1f3f7f860","Type":"ContainerStarted","Data":"247dd803acdeb5c369b678e07c0bdb1ba2f8d897d7034f1c5540f3f7aff29479"}
Oct 31 01:42:01.932926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.932894    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr" event={"ID":"0f6be199-c533-4e82-b569-59bd860268b8","Type":"ContainerStarted","Data":"9766ea1dc401520c40671096ac734d8f8d4b2d9f288f461bb9bc044db3ff6e49"}
Oct 31 01:42:01.933150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.932948    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr" event={"ID":"0f6be199-c533-4e82-b569-59bd860268b8","Type":"ContainerStarted","Data":"aa9821ed58ed98f3a1bc89009e4000354a4959df431422ac8a0f110ee1e48035"}
Oct 31 01:42:01.944207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.944150    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt" podStartSLOduration=1.944110481 podStartE2EDuration="1.944110481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:01.942265948 +0000 UTC m=+1279.890914342" watchObservedRunningTime="2026-10-31 01:42:01.944110481 +0000 UTC m=+1279.892758875"
Oct 31 01:42:01.961721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:01.961668    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr" podStartSLOduration=1.9616524210000001 podStartE2EDuration="1.961652421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:01.958925771 +0000 UTC m=+1279.907574164" watchObservedRunningTime="2026-10-31 01:42:01.961652421 +0000 UTC m=+1279.910300813"
Oct 31 01:42:02.132185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.132159    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:42:02.235610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.235527    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-openshift-ca\") pod \"1c94eb49-dffe-47d8-8777-f1331e88b016\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") "
Oct 31 01:42:02.235610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.235576    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-xks-ca\") pod \"1c94eb49-dffe-47d8-8777-f1331e88b016\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") "
Oct 31 01:42:02.235833 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.235781    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c94eb49-dffe-47d8-8777-f1331e88b016-kube-api-access-xwspm\" (UniqueName: \"kubernetes.io/projected/1c94eb49-dffe-47d8-8777-f1331e88b016-kube-api-access-xwspm\") pod \"1c94eb49-dffe-47d8-8777-f1331e88b016\" (UID: \"1c94eb49-dffe-47d8-8777-f1331e88b016\") "
Oct 31 01:42:02.235887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.235846    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-openshift-ca" pod "1c94eb49-dffe-47d8-8777-f1331e88b016" (UID: "1c94eb49-dffe-47d8-8777-f1331e88b016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:02.237821 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.237786    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:02.247079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.243910    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c94eb49-dffe-47d8-8777-f1331e88b016-kube-api-access-xwspm" pod "1c94eb49-dffe-47d8-8777-f1331e88b016" (UID: "1c94eb49-dffe-47d8-8777-f1331e88b016"). InnerVolumeSpecName "kube-api-access-xwspm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:02.247079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.246667    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-xks-ca" pod "1c94eb49-dffe-47d8-8777-f1331e88b016" (UID: "1c94eb49-dffe-47d8-8777-f1331e88b016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:02.338932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.338899    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwspm\" (UniqueName: \"kubernetes.io/projected/1c94eb49-dffe-47d8-8777-f1331e88b016-kube-api-access-xwspm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:02.338932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.338932    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c94eb49-dffe-47d8-8777-f1331e88b016-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:02.651930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.651896    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:42:02.652110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.651960    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:42:02.940720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.940689    5293 generic.go:356] "Generic (PLEG): container finished" podID="0f6be199-c533-4e82-b569-59bd860268b8" containerID="9766ea1dc401520c40671096ac734d8f8d4b2d9f288f461bb9bc044db3ff6e49" exitCode=0
Oct 31 01:42:02.941186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.940787    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr" event={"ID":"0f6be199-c533-4e82-b569-59bd860268b8","Type":"ContainerDied","Data":"9766ea1dc401520c40671096ac734d8f8d4b2d9f288f461bb9bc044db3ff6e49"}
Oct 31 01:42:02.942301 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.942278    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf" event={"ID":"1c94eb49-dffe-47d8-8777-f1331e88b016","Type":"ContainerDied","Data":"f111c6c60b9654ae8ce625da55de5e646ab1d7824a8a1345916f67256ca81f40"}
Oct 31 01:42:02.942434 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.942304    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"
Oct 31 01:42:02.942434 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.942323    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f111c6c60b9654ae8ce625da55de5e646ab1d7824a8a1345916f67256ca81f40"
Oct 31 01:42:02.944261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.944239    5293 generic.go:356] "Generic (PLEG): container finished" podID="d80f96a4-dabc-4ac6-a225-76e1f3f7f860" containerID="1403b5aa7dc6f16bdb5a60095785923c65a0533504baca896ea8fea92f4f9575" exitCode=0
Oct 31 01:42:02.944359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:02.944302    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt" event={"ID":"d80f96a4-dabc-4ac6-a225-76e1f3f7f860","Type":"ContainerDied","Data":"1403b5aa7dc6f16bdb5a60095785923c65a0533504baca896ea8fea92f4f9575"}
Oct 31 01:42:03.651924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:03.651886    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:42:03.652098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:03.651943    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:42:04.177277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.177254    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:04.181935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.181913    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:04.260486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.260394    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6be199-c533-4e82-b569-59bd860268b8-kube-api-access-jclgv\" (UniqueName: \"kubernetes.io/projected/0f6be199-c533-4e82-b569-59bd860268b8-kube-api-access-jclgv\") pod \"0f6be199-c533-4e82-b569-59bd860268b8\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") "
Oct 31 01:42:04.260486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.260457    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-openshift-ca\") pod \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") "
Oct 31 01:42:04.260717 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.260494    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-openshift-ca\") pod \"0f6be199-c533-4e82-b569-59bd860268b8\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") "
Oct 31 01:42:04.260717 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.260550    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-kube-api-access-lrb46\" (UniqueName: \"kubernetes.io/projected/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-kube-api-access-lrb46\") pod \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") "
Oct 31 01:42:04.260717 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.260604    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-xks-ca\") pod \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\" (UID: \"d80f96a4-dabc-4ac6-a225-76e1f3f7f860\") "
Oct 31 01:42:04.260717 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.260684    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-xks-ca\") pod \"0f6be199-c533-4e82-b569-59bd860268b8\" (UID: \"0f6be199-c533-4e82-b569-59bd860268b8\") "
Oct 31 01:42:04.260933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.260848    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-openshift-ca" pod "d80f96a4-dabc-4ac6-a225-76e1f3f7f860" (UID: "d80f96a4-dabc-4ac6-a225-76e1f3f7f860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:04.260933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.260898    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-openshift-ca" pod "0f6be199-c533-4e82-b569-59bd860268b8" (UID: "0f6be199-c533-4e82-b569-59bd860268b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:04.261264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.261239    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.261368 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.261270    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.263882 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.263830    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6be199-c533-4e82-b569-59bd860268b8-kube-api-access-jclgv" pod "0f6be199-c533-4e82-b569-59bd860268b8" (UID: "0f6be199-c533-4e82-b569-59bd860268b8"). InnerVolumeSpecName "kube-api-access-jclgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:04.264133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.264078    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-xks-ca" pod "d80f96a4-dabc-4ac6-a225-76e1f3f7f860" (UID: "d80f96a4-dabc-4ac6-a225-76e1f3f7f860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:04.264133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.264110    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-kube-api-access-lrb46" pod "d80f96a4-dabc-4ac6-a225-76e1f3f7f860" (UID: "d80f96a4-dabc-4ac6-a225-76e1f3f7f860"). InnerVolumeSpecName "kube-api-access-lrb46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:04.264236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.264166    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-xks-ca" pod "0f6be199-c533-4e82-b569-59bd860268b8" (UID: "0f6be199-c533-4e82-b569-59bd860268b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:04.362548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.362514    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclgv\" (UniqueName: \"kubernetes.io/projected/0f6be199-c533-4e82-b569-59bd860268b8-kube-api-access-jclgv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.362548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.362543    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrb46\" (UniqueName: \"kubernetes.io/projected/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-kube-api-access-lrb46\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.362797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.362557    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80f96a4-dabc-4ac6-a225-76e1f3f7f860-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.362797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.362571    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6be199-c533-4e82-b569-59bd860268b8-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.657073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.657013    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:42:04.657683 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.657662    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:42:04.658814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.658791    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 01:42:04.957103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.957006    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"
Oct 31 01:42:04.957103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.957028    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt" event={"ID":"d80f96a4-dabc-4ac6-a225-76e1f3f7f860","Type":"ContainerDied","Data":"247dd803acdeb5c369b678e07c0bdb1ba2f8d897d7034f1c5540f3f7aff29479"}
Oct 31 01:42:04.957103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.957084    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="247dd803acdeb5c369b678e07c0bdb1ba2f8d897d7034f1c5540f3f7aff29479"
Oct 31 01:42:04.958547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.958528    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"
Oct 31 01:42:04.958547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.958533    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr" event={"ID":"0f6be199-c533-4e82-b569-59bd860268b8","Type":"ContainerDied","Data":"aa9821ed58ed98f3a1bc89009e4000354a4959df431422ac8a0f110ee1e48035"}
Oct 31 01:42:04.958755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:04.958566    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa9821ed58ed98f3a1bc89009e4000354a4959df431422ac8a0f110ee1e48035"
Oct 31 01:42:11.777387 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:42:11.777349    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:42:13.702163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.702120    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"]
Oct 31 01:42:13.702890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.702871    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0f6be199-c533-4e82-b569-59bd860268b8" containerName="revoke-keys"
Oct 31 01:42:13.702939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.702894    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6be199-c533-4e82-b569-59bd860268b8" containerName="revoke-keys"
Oct 31 01:42:13.702939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.702924    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1c94eb49-dffe-47d8-8777-f1331e88b016" containerName="revoke-keys"
Oct 31 01:42:13.702939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.702933    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c94eb49-dffe-47d8-8777-f1331e88b016" containerName="revoke-keys"
Oct 31 01:42:13.703027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.702953    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d80f96a4-dabc-4ac6-a225-76e1f3f7f860" containerName="revoke-keys"
Oct 31 01:42:13.703027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.702961    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80f96a4-dabc-4ac6-a225-76e1f3f7f860" containerName="revoke-keys"
Oct 31 01:42:13.703102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.703091    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6be199-c533-4e82-b569-59bd860268b8" containerName="revoke-keys"
Oct 31 01:42:13.703136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.703108    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c94eb49-dffe-47d8-8777-f1331e88b016" containerName="revoke-keys"
Oct 31 01:42:13.703136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.703122    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80f96a4-dabc-4ac6-a225-76e1f3f7f860" containerName="revoke-keys"
Oct 31 01:42:13.708246 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.708224    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.713545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.713326    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:42:13.713946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.713830    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:42:13.713946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.713883    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:42:13.714139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.713953    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:42:13.716927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.716900    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"]
Oct 31 01:42:13.778477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.778429    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668sd\" (UniqueName: \"kubernetes.io/projected/c1f004ee-b685-4130-879e-9a115968d320-kube-api-access-668sd\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.778627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.778488    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f004ee-b685-4130-879e-9a115968d320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.778627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.778594    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f004ee-b685-4130-879e-9a115968d320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.879923 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.879885    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f004ee-b685-4130-879e-9a115968d320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.880134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.880079    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668sd\" (UniqueName: \"kubernetes.io/projected/c1f004ee-b685-4130-879e-9a115968d320-kube-api-access-668sd\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.880134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.880120    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f004ee-b685-4130-879e-9a115968d320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.881253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.881160    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f004ee-b685-4130-879e-9a115968d320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.906715 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.906071    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f004ee-b685-4130-879e-9a115968d320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:13.939763 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:13.939728    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668sd\" (UniqueName: \"kubernetes.io/projected/c1f004ee-b685-4130-879e-9a115968d320-kube-api-access-668sd\") pod \"maas-api-revoke-sub-6017d23d-p6x98\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:14.028963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:14.028885    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:14.330565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:14.330529    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"]
Oct 31 01:42:14.346232 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:14.339864    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f004ee_b685_4130_879e_9a115968d320.slice/crio-b37de18b3a64b0a44e667995fec5ce29e76b064ca21888b58bf5791779a8c4ea WatchSource:0}: Error finding container b37de18b3a64b0a44e667995fec5ce29e76b064ca21888b58bf5791779a8c4ea: Status 404 returned error can't find the container with id b37de18b3a64b0a44e667995fec5ce29e76b064ca21888b58bf5791779a8c4ea
Oct 31 01:42:15.093132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:15.091317    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98" event={"ID":"c1f004ee-b685-4130-879e-9a115968d320","Type":"ContainerStarted","Data":"3361116c67eafdd05403fb7dab6c1a5183027130efd0a731cc37896f316a93ec"}
Oct 31 01:42:15.093132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:15.091387    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98" event={"ID":"c1f004ee-b685-4130-879e-9a115968d320","Type":"ContainerStarted","Data":"b37de18b3a64b0a44e667995fec5ce29e76b064ca21888b58bf5791779a8c4ea"}
Oct 31 01:42:16.130064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:16.128569    5293 generic.go:356] "Generic (PLEG): container finished" podID="c1f004ee-b685-4130-879e-9a115968d320" containerID="3361116c67eafdd05403fb7dab6c1a5183027130efd0a731cc37896f316a93ec" exitCode=0
Oct 31 01:42:16.130064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:16.128677    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98" event={"ID":"c1f004ee-b685-4130-879e-9a115968d320","Type":"ContainerDied","Data":"3361116c67eafdd05403fb7dab6c1a5183027130efd0a731cc37896f316a93ec"}
Oct 31 01:42:17.289734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.289705    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:17.369119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.368720    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f004ee-b685-4130-879e-9a115968d320-kube-api-access-668sd\" (UniqueName: \"kubernetes.io/projected/c1f004ee-b685-4130-879e-9a115968d320-kube-api-access-668sd\") pod \"c1f004ee-b685-4130-879e-9a115968d320\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") "
Oct 31 01:42:17.369119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.368848    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f004ee-b685-4130-879e-9a115968d320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f004ee-b685-4130-879e-9a115968d320-maas-api-openshift-ca\") pod \"c1f004ee-b685-4130-879e-9a115968d320\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") "
Oct 31 01:42:17.369119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.368954    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f004ee-b685-4130-879e-9a115968d320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f004ee-b685-4130-879e-9a115968d320-maas-api-xks-ca\") pod \"c1f004ee-b685-4130-879e-9a115968d320\" (UID: \"c1f004ee-b685-4130-879e-9a115968d320\") "
Oct 31 01:42:17.369405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.369249    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f004ee-b685-4130-879e-9a115968d320-maas-api-openshift-ca" pod "c1f004ee-b685-4130-879e-9a115968d320" (UID: "c1f004ee-b685-4130-879e-9a115968d320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:17.369460 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.369424    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f004ee-b685-4130-879e-9a115968d320-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:17.371245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.371211    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f004ee-b685-4130-879e-9a115968d320-maas-api-xks-ca" pod "c1f004ee-b685-4130-879e-9a115968d320" (UID: "c1f004ee-b685-4130-879e-9a115968d320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:17.371384 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.371251    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f004ee-b685-4130-879e-9a115968d320-kube-api-access-668sd" pod "c1f004ee-b685-4130-879e-9a115968d320" (UID: "c1f004ee-b685-4130-879e-9a115968d320"). InnerVolumeSpecName "kube-api-access-668sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:17.392829 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.392767    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:42:17.470221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.470185    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668sd\" (UniqueName: \"kubernetes.io/projected/c1f004ee-b685-4130-879e-9a115968d320-kube-api-access-668sd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:17.470221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:17.470206    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f004ee-b685-4130-879e-9a115968d320-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:18.172349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:18.172311    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"
Oct 31 01:42:18.193483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:18.193447    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98" event={"ID":"c1f004ee-b685-4130-879e-9a115968d320","Type":"ContainerDied","Data":"b37de18b3a64b0a44e667995fec5ce29e76b064ca21888b58bf5791779a8c4ea"}
Oct 31 01:42:18.193652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:18.193513    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37de18b3a64b0a44e667995fec5ce29e76b064ca21888b58bf5791779a8c4ea"
Oct 31 01:42:23.936062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.936003    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"]
Oct 31 01:42:23.936511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.936498    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c1f004ee-b685-4130-879e-9a115968d320" containerName="revoke-keys"
Oct 31 01:42:23.936550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.936512    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f004ee-b685-4130-879e-9a115968d320" containerName="revoke-keys"
Oct 31 01:42:23.936607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.936598    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f004ee-b685-4130-879e-9a115968d320" containerName="revoke-keys"
Oct 31 01:42:23.941055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.941026    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:23.947056 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.947004    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:42:23.947596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.947528    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:42:23.947596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.947554    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:42:23.947768 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.947598    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:42:23.950834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:23.950807    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"]
Oct 31 01:42:24.033502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.033463    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hf4f\" (UniqueName: \"kubernetes.io/projected/63a33b0f-8152-4a2f-8175-61da9554652c-kube-api-access-6hf4f\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.033687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.033557    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.033687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.033621    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.135195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.135157    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hf4f\" (UniqueName: \"kubernetes.io/projected/63a33b0f-8152-4a2f-8175-61da9554652c-kube-api-access-6hf4f\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.135393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.135228    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.135393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.135263    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.135828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.135807    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.137779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.137752    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.151067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.151007    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hf4f\" (UniqueName: \"kubernetes.io/projected/63a33b0f-8152-4a2f-8175-61da9554652c-kube-api-access-6hf4f\") pod \"maas-api-revoke-sub-c6c8afce-m6mst\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.256165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.256077    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:24.547499 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:24.547427    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"]
Oct 31 01:42:24.550899 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:24.550866    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a33b0f_8152_4a2f_8175_61da9554652c.slice/crio-2fbb2a3af6b3b0b775e6f094c33dc2283d3a1b7cbe93cd21c6dd9bf0c58ad24c WatchSource:0}: Error finding container 2fbb2a3af6b3b0b775e6f094c33dc2283d3a1b7cbe93cd21c6dd9bf0c58ad24c: Status 404 returned error can't find the container with id 2fbb2a3af6b3b0b775e6f094c33dc2283d3a1b7cbe93cd21c6dd9bf0c58ad24c
Oct 31 01:42:25.214288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:25.214248    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst" event={"ID":"63a33b0f-8152-4a2f-8175-61da9554652c","Type":"ContainerStarted","Data":"8948b79ed9a166fe9d42386a63da192ac48f0c84391488e1815eee70f2b2db4b"}
Oct 31 01:42:25.214686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:25.214317    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst" event={"ID":"63a33b0f-8152-4a2f-8175-61da9554652c","Type":"ContainerStarted","Data":"2fbb2a3af6b3b0b775e6f094c33dc2283d3a1b7cbe93cd21c6dd9bf0c58ad24c"}
Oct 31 01:42:25.261758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:25.261707    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst" podStartSLOduration=2.261693994 podStartE2EDuration="2.261693994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:25.259682785 +0000 UTC m=+1303.208331178" watchObservedRunningTime="2026-10-31 01:42:25.261693994 +0000 UTC m=+1303.210342418"
Oct 31 01:42:26.222761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:26.222730    5293 generic.go:356] "Generic (PLEG): container finished" podID="63a33b0f-8152-4a2f-8175-61da9554652c" containerID="8948b79ed9a166fe9d42386a63da192ac48f0c84391488e1815eee70f2b2db4b" exitCode=0
Oct 31 01:42:26.223200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:26.222824    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst" event={"ID":"63a33b0f-8152-4a2f-8175-61da9554652c","Type":"ContainerDied","Data":"8948b79ed9a166fe9d42386a63da192ac48f0c84391488e1815eee70f2b2db4b"}
Oct 31 01:42:27.389105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.389084    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:27.486117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.481399    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a33b0f-8152-4a2f-8175-61da9554652c-kube-api-access-6hf4f\" (UniqueName: \"kubernetes.io/projected/63a33b0f-8152-4a2f-8175-61da9554652c-kube-api-access-6hf4f\") pod \"63a33b0f-8152-4a2f-8175-61da9554652c\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") "
Oct 31 01:42:27.486117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.481467    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-openshift-ca\") pod \"63a33b0f-8152-4a2f-8175-61da9554652c\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") "
Oct 31 01:42:27.486117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.481517    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-xks-ca\") pod \"63a33b0f-8152-4a2f-8175-61da9554652c\" (UID: \"63a33b0f-8152-4a2f-8175-61da9554652c\") "
Oct 31 01:42:27.487576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.487528    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-openshift-ca" pod "63a33b0f-8152-4a2f-8175-61da9554652c" (UID: "63a33b0f-8152-4a2f-8175-61da9554652c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:27.517779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.517746    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a33b0f-8152-4a2f-8175-61da9554652c-kube-api-access-6hf4f" pod "63a33b0f-8152-4a2f-8175-61da9554652c" (UID: "63a33b0f-8152-4a2f-8175-61da9554652c"). InnerVolumeSpecName "kube-api-access-6hf4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:27.521428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.521402    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-xks-ca" pod "63a33b0f-8152-4a2f-8175-61da9554652c" (UID: "63a33b0f-8152-4a2f-8175-61da9554652c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:27.595414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.593056    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hf4f\" (UniqueName: \"kubernetes.io/projected/63a33b0f-8152-4a2f-8175-61da9554652c-kube-api-access-6hf4f\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:27.595414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.593098    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:27.595414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:27.593114    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a33b0f-8152-4a2f-8175-61da9554652c-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:28.241854 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:28.241812    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst" event={"ID":"63a33b0f-8152-4a2f-8175-61da9554652c","Type":"ContainerDied","Data":"2fbb2a3af6b3b0b775e6f094c33dc2283d3a1b7cbe93cd21c6dd9bf0c58ad24c"}
Oct 31 01:42:28.242021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:28.241889    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fbb2a3af6b3b0b775e6f094c33dc2283d3a1b7cbe93cd21c6dd9bf0c58ad24c"
Oct 31 01:42:28.242021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:28.241979    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"
Oct 31 01:42:30.344988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.344949    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"]
Oct 31 01:42:30.345714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.345697    5293 cpu_manager.go:414] "containerMap: removing container" podUID="63a33b0f-8152-4a2f-8175-61da9554652c" containerName="revoke-keys"
Oct 31 01:42:30.345808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.345715    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a33b0f-8152-4a2f-8175-61da9554652c" containerName="revoke-keys"
Oct 31 01:42:30.345870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.345817    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a33b0f-8152-4a2f-8175-61da9554652c" containerName="revoke-keys"
Oct 31 01:42:30.350481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.350459    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.356001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.355848    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:42:30.356001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.355939    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:42:30.356196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.356017    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:42:30.356196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.356103    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:42:30.356858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.356833    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"]
Oct 31 01:42:30.450388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.450356    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.450540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.450419    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.450540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.450514    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bz7\" (UniqueName: \"kubernetes.io/projected/bb84b66e-25dd-4bc2-9a95-024ab80352b5-kube-api-access-n4bz7\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.551232 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.551198    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.551399 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.551245    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.551399 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.551274    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bz7\" (UniqueName: \"kubernetes.io/projected/bb84b66e-25dd-4bc2-9a95-024ab80352b5-kube-api-access-n4bz7\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.551849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.551825    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.553814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.553786    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.564695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.564671    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bz7\" (UniqueName: \"kubernetes.io/projected/bb84b66e-25dd-4bc2-9a95-024ab80352b5-kube-api-access-n4bz7\") pod \"maas-api-revoke-sub-17ee28f9-49gcd\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.675701 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.675669    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:30.958733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:30.958648    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"]
Oct 31 01:42:30.962431 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:30.962404    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb84b66e_25dd_4bc2_9a95_024ab80352b5.slice/crio-7fa8b7c313297c9e73fbba9579fbe05a24ee876ad3737e8c506d9683d452be9c WatchSource:0}: Error finding container 7fa8b7c313297c9e73fbba9579fbe05a24ee876ad3737e8c506d9683d452be9c: Status 404 returned error can't find the container with id 7fa8b7c313297c9e73fbba9579fbe05a24ee876ad3737e8c506d9683d452be9c
Oct 31 01:42:31.274596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:31.274560    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd" event={"ID":"bb84b66e-25dd-4bc2-9a95-024ab80352b5","Type":"ContainerStarted","Data":"7fa8b7c313297c9e73fbba9579fbe05a24ee876ad3737e8c506d9683d452be9c"}
Oct 31 01:42:32.282315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:32.282216    5293 generic.go:356] "Generic (PLEG): container finished" podID="bb84b66e-25dd-4bc2-9a95-024ab80352b5" containerID="88ad6703b9b2e59556d58a25d57b46804762c736e95449ae5643922ef48e3260" exitCode=0
Oct 31 01:42:32.282315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:32.282308    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd" event={"ID":"bb84b66e-25dd-4bc2-9a95-024ab80352b5","Type":"ContainerDied","Data":"88ad6703b9b2e59556d58a25d57b46804762c736e95449ae5643922ef48e3260"}
Oct 31 01:42:33.460148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.460124    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:33.584529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.584439    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-xks-ca\") pod \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") "
Oct 31 01:42:33.584709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.584628    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-openshift-ca\") pod \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") "
Oct 31 01:42:33.584709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.584653    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb84b66e-25dd-4bc2-9a95-024ab80352b5-kube-api-access-n4bz7\" (UniqueName: \"kubernetes.io/projected/bb84b66e-25dd-4bc2-9a95-024ab80352b5-kube-api-access-n4bz7\") pod \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\" (UID: \"bb84b66e-25dd-4bc2-9a95-024ab80352b5\") "
Oct 31 01:42:33.585070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.585014    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-openshift-ca" pod "bb84b66e-25dd-4bc2-9a95-024ab80352b5" (UID: "bb84b66e-25dd-4bc2-9a95-024ab80352b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:33.586654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.586633    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-xks-ca" pod "bb84b66e-25dd-4bc2-9a95-024ab80352b5" (UID: "bb84b66e-25dd-4bc2-9a95-024ab80352b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:33.586927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.586905    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb84b66e-25dd-4bc2-9a95-024ab80352b5-kube-api-access-n4bz7" pod "bb84b66e-25dd-4bc2-9a95-024ab80352b5" (UID: "bb84b66e-25dd-4bc2-9a95-024ab80352b5"). InnerVolumeSpecName "kube-api-access-n4bz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:33.686412 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.686383    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:33.686412 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.686408    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb84b66e-25dd-4bc2-9a95-024ab80352b5-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:33.686412 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:33.686419    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bz7\" (UniqueName: \"kubernetes.io/projected/bb84b66e-25dd-4bc2-9a95-024ab80352b5-kube-api-access-n4bz7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:34.295788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:34.295755    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"
Oct 31 01:42:34.295979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:34.295785    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd" event={"ID":"bb84b66e-25dd-4bc2-9a95-024ab80352b5","Type":"ContainerDied","Data":"7fa8b7c313297c9e73fbba9579fbe05a24ee876ad3737e8c506d9683d452be9c"}
Oct 31 01:42:34.295979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:34.295836    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fa8b7c313297c9e73fbba9579fbe05a24ee876ad3737e8c506d9683d452be9c"
Oct 31 01:42:34.582670 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:34.582589    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:42:38.114843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.114792    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"]
Oct 31 01:42:38.145478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.145443    5293 cpu_manager.go:414] "containerMap: removing container" podUID="bb84b66e-25dd-4bc2-9a95-024ab80352b5" containerName="revoke-keys"
Oct 31 01:42:38.145678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.145661    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb84b66e-25dd-4bc2-9a95-024ab80352b5" containerName="revoke-keys"
Oct 31 01:42:38.145982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.145964    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb84b66e-25dd-4bc2-9a95-024ab80352b5" containerName="revoke-keys"
Oct 31 01:42:38.155707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.155680    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"]
Oct 31 01:42:38.156095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.156077    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.286793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.286749    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.293137 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.293102    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff97f77-3447-4ff9-91ea-d7dda29aa919-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.293854 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.293826    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.294124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.294104    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.294323 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.294305    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.294498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.294480    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5zh\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-kube-api-access-zg5zh\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.294685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.294667    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.294839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.294824    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.295028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.295012    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.295358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.291968    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"]
Oct 31 01:42:38.383310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.383171    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"]
Oct 31 01:42:38.383715 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.383664    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.434256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.434223    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff97f77-3447-4ff9-91ea-d7dda29aa919-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.434509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.434490    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.434820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.434796    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.434988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.434971    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.435152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.435126    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b6fc1f12130d\""
Oct 31 01:42:38.446790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.446767    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 01:42:38.460961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.436125    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff97f77-3447-4ff9-91ea-d7dda29aa919-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.461164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.436370    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.461349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.436615    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.461561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.436952    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 01:42:38.461692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.437097    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-h627d\""
Oct 31 01:42:38.461876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.437162    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 01:42:38.462073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.435128    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5zh\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-kube-api-access-zg5zh\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.462310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.462292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.462548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.459971    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 01:42:38.464782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.463807    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.464782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.462531    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.471416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.469687    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.471416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.469791    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.472754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.471761    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.503011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.483816    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.503011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.484212    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5zh\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-kube-api-access-zg5zh\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.503011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.485092    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.503011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.485512    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.524059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.522346    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:38.579080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.578656    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-plugins-config-volume\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.579080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.578724    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-routing-overlay-volume\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.579080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.578870    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnl5p\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-kube-api-access-cnl5p\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.579080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.578911    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-provider-credentials\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.680497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.680456    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnl5p\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-kube-api-access-cnl5p\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.680642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.680526    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-provider-credentials\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.680642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.680582    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-plugins-config-volume\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.680642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.680624    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-routing-overlay-volume\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.682714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.681577    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-routing-overlay-volume\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.683075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.683049    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-plugins-config-volume\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.707810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.706525    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-provider-credentials\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.708233 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.708205    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnl5p\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-kube-api-access-cnl5p\") pod \"payload-processing-external-model-f48c669b9-ckz8g\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") " pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.785264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.785229    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:38.905337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.905303    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"]
Oct 31 01:42:38.908716 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:38.908688    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff97f77_3447_4ff9_91ea_d7dda29aa919.slice/crio-e175f075c14aaca41a5bc969c730c216a7cb7666236168dd006dab113bea5453 WatchSource:0}: Error finding container e175f075c14aaca41a5bc969c730c216a7cb7666236168dd006dab113bea5453: Status 404 returned error can't find the container with id e175f075c14aaca41a5bc969c730c216a7cb7666236168dd006dab113bea5453
Oct 31 01:42:38.911729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.911695    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:38.911808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.911769    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:38.911808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:38.911802    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:39.072487 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.072454    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"]
Oct 31 01:42:39.074064 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:39.074019    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db518e4_7e54_4bf4_a0a9_8487b3570264.slice/crio-591cbd0dde1e391f79efd7a46735101f6994ca8c5c27c7700e1b4269b510aabb WatchSource:0}: Error finding container 591cbd0dde1e391f79efd7a46735101f6994ca8c5c27c7700e1b4269b510aabb: Status 404 returned error can't find the container with id 591cbd0dde1e391f79efd7a46735101f6994ca8c5c27c7700e1b4269b510aabb
Oct 31 01:42:39.492957 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.492923    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" event={"ID":"aff97f77-3447-4ff9-91ea-d7dda29aa919","Type":"ContainerStarted","Data":"0e784058d05de146c9746571856f4fad7c8cefd69d8dc3bff3a33da96bc6ce6c"}
Oct 31 01:42:39.493382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.492987    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" event={"ID":"aff97f77-3447-4ff9-91ea-d7dda29aa919","Type":"ContainerStarted","Data":"e175f075c14aaca41a5bc969c730c216a7cb7666236168dd006dab113bea5453"}
Oct 31 01:42:39.494686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.494643    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g" event={"ID":"0db518e4-7e54-4bf4-a0a9-8487b3570264","Type":"ContainerStarted","Data":"0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b"}
Oct 31 01:42:39.494815 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.494700    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g" event={"ID":"0db518e4-7e54-4bf4-a0a9-8487b3570264","Type":"ContainerStarted","Data":"591cbd0dde1e391f79efd7a46735101f6994ca8c5c27c7700e1b4269b510aabb"}
Oct 31 01:42:39.494815 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.494735    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:39.525064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.524062    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:39.566491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.566435    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" podStartSLOduration=1.56642073 podStartE2EDuration="1.56642073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:39.564065894 +0000 UTC m=+1317.512714280" watchObservedRunningTime="2026-10-31 01:42:39.56642073 +0000 UTC m=+1317.515069121"
Oct 31 01:42:39.588969 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:39.588851    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g" podStartSLOduration=1.588801531 podStartE2EDuration="1.588801531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:39.587947232 +0000 UTC m=+1317.536595624" watchObservedRunningTime="2026-10-31 01:42:39.588801531 +0000 UTC m=+1317.537449915"
Oct 31 01:42:40.524890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:40.524859    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:42:40.525490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:40.525448    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 01:42:41.525549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:41.525502    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:42:41.526077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:41.525561    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 01:42:41.783116 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:42:41.780337    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:42:42.523763 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:42.523664    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:42:42.523763 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:42.523746    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 01:42:43.525064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:43.524379    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:42:43.525064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:43.524453    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 01:42:44.527515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:44.527476    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:42:44.527956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:44.527558    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 01:42:45.091147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.087864    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"]
Oct 31 01:42:45.093730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.093705    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.119348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.119323    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"]
Oct 31 01:42:45.185549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.185517    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.185714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.185557    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.185714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.185575    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nt82\" (UniqueName: \"kubernetes.io/projected/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-kube-api-access-2nt82\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.185714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.185699    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.185963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.185928    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.186223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.186198    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.186388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.186366    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.186449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.186430    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.186601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.186580    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287373    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287425    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287474    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287551    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287580    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287616    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287676    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287715    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.287819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287744    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nt82\" (UniqueName: \"kubernetes.io/projected/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-kube-api-access-2nt82\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.288034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287830    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.288034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.287907    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.288213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.288189    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.288264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.288248    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.288508 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.288489    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.289992 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.289962    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.290531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.290510    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.300651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.300073    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.300651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.300588    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nt82\" (UniqueName: \"kubernetes.io/projected/d67d9b4f-a0df-43a7-97b7-88df20e0aa19-kube-api-access-2nt82\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v46nb\" (UID: \"d67d9b4f-a0df-43a7-97b7-88df20e0aa19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.425477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.425184    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:45.723910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.723880    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"]
Oct 31 01:42:45.724589 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:45.724555    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67d9b4f_a0df_43a7_97b7_88df20e0aa19.slice/crio-cc8a9d6504268a9b612f150f88b1844148cc1cd60136c6eca31a1e08997f2252 WatchSource:0}: Error finding container cc8a9d6504268a9b612f150f88b1844148cc1cd60136c6eca31a1e08997f2252: Status 404 returned error can't find the container with id cc8a9d6504268a9b612f150f88b1844148cc1cd60136c6eca31a1e08997f2252
Oct 31 01:42:45.728142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.727742    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:45.728142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.727836    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:45.728142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.727880    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:45.752691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.752665    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:45.753655 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.753638    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:45.755145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:45.755121    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:42:46.565513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:46.565476    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" event={"ID":"d67d9b4f-a0df-43a7-97b7-88df20e0aa19","Type":"ContainerStarted","Data":"0456f0729aa93eb437006c6895c41f00f870d4edc7f9ccee39305cddb27b40d6"}
Oct 31 01:42:46.565720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:46.565531    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" event={"ID":"d67d9b4f-a0df-43a7-97b7-88df20e0aa19","Type":"ContainerStarted","Data":"cc8a9d6504268a9b612f150f88b1844148cc1cd60136c6eca31a1e08997f2252"}
Oct 31 01:42:46.592738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:46.592686    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podStartSLOduration=1.5926704059999999 podStartE2EDuration="1.592670406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:46.589483836 +0000 UTC m=+1324.538132238" watchObservedRunningTime="2026-10-31 01:42:46.592670406 +0000 UTC m=+1324.541318798"
Oct 31 01:42:47.216443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.216400    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"]
Oct 31 01:42:47.230536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.230512    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.248426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.248401    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"distinct-w1-0e8b5c-kserve-self-signed-certs\""
Oct 31 01:42:47.319685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.318109    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"]
Oct 31 01:42:47.351363 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.351331    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tmp-dir\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.351669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.351647    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-dshm\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.364065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.351795    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-home\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.364364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.364337    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-model-cache\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.364567 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.364548    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhq4x\" (UniqueName: \"kubernetes.io/projected/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-kube-api-access-mhq4x\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.364754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.364735    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tls-certs\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.430266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.430170    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:47.465509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.460987    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:47.465509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.461070    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:42:47.479409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.474237    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tls-certs\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.479409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.474337    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tmp-dir\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.479409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.474407    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-dshm\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.479409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.474444    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-home\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.479409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.474476    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-model-cache\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.479409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.474527    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhq4x\" (UniqueName: \"kubernetes.io/projected/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-kube-api-access-mhq4x\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.491295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.482031    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-home\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.491295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.482990    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tmp-dir\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.519373 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.491826    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-model-cache\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.564984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.564802    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhq4x\" (UniqueName: \"kubernetes.io/projected/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-kube-api-access-mhq4x\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.567362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.565497    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-dshm\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.580269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.580249    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tls-certs\") pod \"distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:47.773708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:47.773595    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:48.444068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:48.435451    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:48.444068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:48.435536    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:42:48.836170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:48.836139    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"]
Oct 31 01:42:48.836886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:48.836858    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" event={"ID":"4d52d8a0-8ca2-4d61-91f1-e992abdbe884","Type":"ContainerStarted","Data":"8e9449745eb56ebd6cac969f5ff24ee22f44feeae4e9ebf887a8332572365f7a"}
Oct 31 01:42:49.429066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:49.428358    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:49.429066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:49.428417    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:42:49.873068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:49.871141    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" event={"ID":"4d52d8a0-8ca2-4d61-91f1-e992abdbe884","Type":"ContainerStarted","Data":"c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7"}
Oct 31 01:42:49.891328 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:49.891270    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podStartSLOduration=2.891253233 podStartE2EDuration="2.891253233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:49.890076217 +0000 UTC m=+1327.838724610" watchObservedRunningTime="2026-10-31 01:42:49.891253233 +0000 UTC m=+1327.839901626"
Oct 31 01:42:50.427065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:50.426373    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:50.427065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:50.426435    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:42:50.540065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:50.539254    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:42:50.777111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:50.776187    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:51.426818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:51.426393    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:51.426818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:51.426446    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:42:52.425924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:52.425890    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:52.426124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:52.425950    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:42:53.426381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:53.426342    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:53.427073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:53.426996    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:42:54.791239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:54.791210    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:42:55.426447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:55.426418    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:55.426615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:55.426470    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:56.425066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.421594    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"]
Oct 31 01:42:56.473200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.473156    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:56.473499 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.473465    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:56.540701 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.540579    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.657880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.657840    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"]
Oct 31 01:42:56.699887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.699808    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.700158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.700138    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.700284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.700268    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.700402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.700389    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.700508 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.700495    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.700609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.700596    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a38bc9-35da-45db-b4c4-18a32e1b5041-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.700782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.700768    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.700888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.700875    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.701009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.700996    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhvn5\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-kube-api-access-vhvn5\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.802514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802470    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.802514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802531    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.802810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802581    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhvn5\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-kube-api-access-vhvn5\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.802810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802700    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.802810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802743    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.802810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802768    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.802810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802802    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.803143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802830    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.803143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.802858    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a38bc9-35da-45db-b4c4-18a32e1b5041-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.803414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.803387    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.803758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.803733    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a38bc9-35da-45db-b4c4-18a32e1b5041-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.804302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.804141    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.804302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.804263    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.804764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.804746    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.806849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.806791    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.806849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.806796    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.815533 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.815476    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.816839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.816814    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhvn5\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-kube-api-access-vhvn5\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:56.968967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:56.968885    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:57.266574 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.266530    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"]
Oct 31 01:42:57.269195 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:42:57.269166    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a38bc9_35da_45db_b4c4_18a32e1b5041.slice/crio-fd30c54257a1c1b2ccb83b20cd86fd2cc43826334170ea97ba3c1cc1ff5a2f04 WatchSource:0}: Error finding container fd30c54257a1c1b2ccb83b20cd86fd2cc43826334170ea97ba3c1cc1ff5a2f04: Status 404 returned error can't find the container with id fd30c54257a1c1b2ccb83b20cd86fd2cc43826334170ea97ba3c1cc1ff5a2f04
Oct 31 01:42:57.271824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.271784    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:57.271906 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.271880    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:57.271955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.271934    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:42:57.518219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.518132    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v46nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:57.519395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.518744    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb" podUID="d67d9b4f-a0df-43a7-97b7-88df20e0aa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:57.531171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.528340    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:57.531171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.529651    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:57.531171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.530838    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v46nb"
Oct 31 01:42:57.774613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.774530    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:57.787640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.787613    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:57.989394 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.989355    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" event={"ID":"75a38bc9-35da-45db-b4c4-18a32e1b5041","Type":"ContainerStarted","Data":"1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a"}
Oct 31 01:42:57.989592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:57.989410    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" event={"ID":"75a38bc9-35da-45db-b4c4-18a32e1b5041","Type":"ContainerStarted","Data":"fd30c54257a1c1b2ccb83b20cd86fd2cc43826334170ea97ba3c1cc1ff5a2f04"}
Oct 31 01:42:58.007088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:58.001931    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:42:58.013118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:58.013065    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podStartSLOduration=2.013029882 podStartE2EDuration="2.013029882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:58.009575371 +0000 UTC m=+1335.958223763" watchObservedRunningTime="2026-10-31 01:42:58.013029882 +0000 UTC m=+1335.961678275"
Oct 31 01:42:58.974241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:58.974127    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:42:58.974839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:58.974793    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:42:58.974985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:58.974852    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:42:59.969568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:59.969531    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:42:59.969730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:42:59.969590    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:43:00.981068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:00.970387    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:43:00.981068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:00.970458    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:43:01.969691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:01.969652    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:43:01.969885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:01.969718    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:43:02.984060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:02.981524    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:43:02.984060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:02.981597    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:43:03.969473 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:03.969439    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:43:03.969640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:03.969491    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:43:04.973318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:04.973290    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:43:04.973724 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:04.973582    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:43:04.974334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:04.974311    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 01:43:09.472066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:09.468457    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:43:09.558226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:09.558188    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:09.559087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:09.559032    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:11.823132 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:43:11.823096    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:43:13.882395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:13.882336    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"]
Oct 31 01:43:13.887000 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:13.886978    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:13.896477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:13.896455    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"]
Oct 31 01:43:13.896727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:13.896708    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"distinct-2-w1-0e8b5c-kserve-self-signed-certs\""
Oct 31 01:43:14.034166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.034129    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tls-certs\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.034505 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.034480    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tmp-dir\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.034738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.034723    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-model-cache\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.034873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.034858    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-home\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.039273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.039227    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzj8j\" (UniqueName: \"kubernetes.io/projected/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-kube-api-access-wzj8j\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.039457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.039439    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-dshm\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.141775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.141687    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-model-cache\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.142047 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.142015    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-home\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.142174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.142080    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzj8j\" (UniqueName: \"kubernetes.io/projected/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-kube-api-access-wzj8j\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.142174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.142113    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-dshm\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.142291 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.142265    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tls-certs\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.142339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.142320    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tmp-dir\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.142391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.142334    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-model-cache\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.142445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.142395    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-home\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.142601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.142581    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tmp-dir\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.144756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.144729    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-dshm\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.144976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.144958    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tls-certs\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.154314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.154275    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzj8j\" (UniqueName: \"kubernetes.io/projected/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-kube-api-access-wzj8j\") pod \"distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") " pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.203497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.203468    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:14.232853 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.232817    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"]
Oct 31 01:43:14.516609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.514803    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"]
Oct 31 01:43:14.907285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:14.907257    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:43:15.215268 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:15.215174    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" event={"ID":"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5","Type":"ContainerStarted","Data":"1d082e476327131484ed2f7462af756e23317fbf0eb4b3c4deafbd50e8f28c5d"}
Oct 31 01:43:15.215268 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:15.215229    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" event={"ID":"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5","Type":"ContainerStarted","Data":"c995425c372f140a788bf17a9676087484706ec29c26aeaf5da15076a4b1f59d"}
Oct 31 01:43:16.203924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:16.203885    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:19.234105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.234028    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g" podUID="0db518e4-7e54-4bf4-a0a9-8487b3570264" containerName="payload-processing" containerID="cri-o://0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b" gracePeriod=25
Oct 31 01:43:19.512193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.512167    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:43:19.532846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.532799    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podStartSLOduration=6.532782547 podStartE2EDuration="6.532782547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:15.240595037 +0000 UTC m=+1353.189243430" watchObservedRunningTime="2026-10-31 01:43:19.532782547 +0000 UTC m=+1357.481430940"
Oct 31 01:43:19.674436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.673380    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-kube-api-access-cnl5p\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-kube-api-access-cnl5p\") pod \"0db518e4-7e54-4bf4-a0a9-8487b3570264\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") "
Oct 31 01:43:19.674436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.673479    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-routing-overlay-volume\") pod \"0db518e4-7e54-4bf4-a0a9-8487b3570264\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") "
Oct 31 01:43:19.674436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.673537    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-provider-credentials\") pod \"0db518e4-7e54-4bf4-a0a9-8487b3570264\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") "
Oct 31 01:43:19.674436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.673685    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-plugins-config-volume\") pod \"0db518e4-7e54-4bf4-a0a9-8487b3570264\" (UID: \"0db518e4-7e54-4bf4-a0a9-8487b3570264\") "
Oct 31 01:43:19.675007 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.674971    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-plugins-config-volume" pod "0db518e4-7e54-4bf4-a0a9-8487b3570264" (UID: "0db518e4-7e54-4bf4-a0a9-8487b3570264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:19.679238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.679018    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-kube-api-access-cnl5p" pod "0db518e4-7e54-4bf4-a0a9-8487b3570264" (UID: "0db518e4-7e54-4bf4-a0a9-8487b3570264"). InnerVolumeSpecName "kube-api-access-cnl5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:19.679739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.679476    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-routing-overlay-volume" pod "0db518e4-7e54-4bf4-a0a9-8487b3570264" (UID: "0db518e4-7e54-4bf4-a0a9-8487b3570264"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:19.696239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.693559    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-provider-credentials" pod "0db518e4-7e54-4bf4-a0a9-8487b3570264" (UID: "0db518e4-7e54-4bf4-a0a9-8487b3570264"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:19.774949 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.774853    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:19.774949 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.774886    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:19.774949 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.774898    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnl5p\" (UniqueName: \"kubernetes.io/projected/0db518e4-7e54-4bf4-a0a9-8487b3570264-kube-api-access-cnl5p\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:19.774949 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:19.774906    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0db518e4-7e54-4bf4-a0a9-8487b3570264-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:20.254813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.254778    5293 generic.go:356] "Generic (PLEG): container finished" podID="0db518e4-7e54-4bf4-a0a9-8487b3570264" containerID="0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b" exitCode=0
Oct 31 01:43:20.255327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.254832    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"
Oct 31 01:43:20.255327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.254866    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g" event={"ID":"0db518e4-7e54-4bf4-a0a9-8487b3570264","Type":"ContainerDied","Data":"0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b"}
Oct 31 01:43:20.255327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.254920    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g" event={"ID":"0db518e4-7e54-4bf4-a0a9-8487b3570264","Type":"ContainerDied","Data":"591cbd0dde1e391f79efd7a46735101f6994ca8c5c27c7700e1b4269b510aabb"}
Oct 31 01:43:20.255327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.254941    5293 scope.go:122] "RemoveContainer" containerID="0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b"
Oct 31 01:43:20.264922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.264902    5293 scope.go:122] "RemoveContainer" containerID="0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b"
Oct 31 01:43:20.265302 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:43:20.265278    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b\": container with ID starting with 0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b not found: ID does not exist" containerID="0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b"
Oct 31 01:43:20.265358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.265309    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b"} err="failed to get container status \"0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b\": rpc error: code = NotFound desc = could not find container \"0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b\": container with ID starting with 0d9e68dbe9e47481e3bfb4756643197a78c9b970eeab192ab67d442dbe00b31b not found: ID does not exist"
Oct 31 01:43:20.271093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.271069    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"]
Oct 31 01:43:20.275676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:20.275657    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f48c669b9-ckz8g"]
Oct 31 01:43:22.178051 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:22.177966    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db518e4-7e54-4bf4-a0a9-8487b3570264" path="/var/lib/kubelet/pods/0db518e4-7e54-4bf4-a0a9-8487b3570264/volumes"
Oct 31 01:43:23.219913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.219876    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"]
Oct 31 01:43:23.220631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.220611    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0db518e4-7e54-4bf4-a0a9-8487b3570264" containerName="payload-processing"
Oct 31 01:43:23.220702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.220634    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db518e4-7e54-4bf4-a0a9-8487b3570264" containerName="payload-processing"
Oct 31 01:43:23.220809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.220797    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db518e4-7e54-4bf4-a0a9-8487b3570264" containerName="payload-processing"
Oct 31 01:43:23.225927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.225906    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.229913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.229894    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:23.230114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.230080    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:23.230226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.229991    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:23.230226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.229985    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:43:23.233785 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.233762    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"]
Oct 31 01:43:23.334229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.334188    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.334658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.334632    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjv2z\" (UniqueName: \"kubernetes.io/projected/0de343be-256a-4503-a8b4-55adc17989a0-kube-api-access-vjv2z\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.334874 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.334857    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.436123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.436090    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.436296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.436154    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjv2z\" (UniqueName: \"kubernetes.io/projected/0de343be-256a-4503-a8b4-55adc17989a0-kube-api-access-vjv2z\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.436296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.436207    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.436977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.436948    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.439223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.439200    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.449263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.449237    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjv2z\" (UniqueName: \"kubernetes.io/projected/0de343be-256a-4503-a8b4-55adc17989a0-kube-api-access-vjv2z\") pod \"maas-api-revoke-sub-936429de-mt8zc\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.545155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.545067    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:23.964286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:23.964246    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"]
Oct 31 01:43:23.966762 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:43:23.966729    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de343be_256a_4503_a8b4_55adc17989a0.slice/crio-2b5af7305093ab622ec4ceed65345dff610d046ff7602023c7568cd161cc4aba WatchSource:0}: Error finding container 2b5af7305093ab622ec4ceed65345dff610d046ff7602023c7568cd161cc4aba: Status 404 returned error can't find the container with id 2b5af7305093ab622ec4ceed65345dff610d046ff7602023c7568cd161cc4aba
Oct 31 01:43:24.206988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.204138    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:24.227408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.227116    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:24.282802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.282769    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc" event={"ID":"0de343be-256a-4503-a8b4-55adc17989a0","Type":"ContainerStarted","Data":"2b5af7305093ab622ec4ceed65345dff610d046ff7602023c7568cd161cc4aba"}
Oct 31 01:43:24.295696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.295666    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:24.546116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.540555    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"]
Oct 31 01:43:24.546116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.545544    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.599758    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"]
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600513    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600560    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600670    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600694    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600752    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600873    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600904    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600948    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.606086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.600983    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjp8\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-kube-api-access-8rjp8\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.702476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702417    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.702476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702472    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.702743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702518    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.702743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702553    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjp8\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-kube-api-access-8rjp8\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.702743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702587    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.702743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702623    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.702743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702731    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.703020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702759    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.703020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.702825    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.704099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.703917    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.704099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.703954    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.704594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.704544    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.704825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.704721    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.711805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.710028    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.711805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.711538    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.712485 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.712441    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.724108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.718785    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.724108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.723185    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjp8\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-kube-api-access-8rjp8\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:24.939070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:24.939000    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:25.289704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:25.289283    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:25.289704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:25.289431    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:25.289704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:25.289519    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:25.313641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:25.313612    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"]
Oct 31 01:43:25.316648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:25.316617    5293 generic.go:356] "Generic (PLEG): container finished" podID="0de343be-256a-4503-a8b4-55adc17989a0" containerID="af3bf6abdb46239fd6047fb30e89cdcaefe5e43ff1aed96d7febcd20bbfbbfc9" exitCode=0
Oct 31 01:43:25.316870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:25.316846    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc" event={"ID":"0de343be-256a-4503-a8b4-55adc17989a0","Type":"ContainerDied","Data":"af3bf6abdb46239fd6047fb30e89cdcaefe5e43ff1aed96d7febcd20bbfbbfc9"}
Oct 31 01:43:25.318594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:25.318566    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" event={"ID":"248a307a-a43d-4d2c-aade-59fb8e0efc2a","Type":"ContainerStarted","Data":"ccc2749ebf3fb5c2827c4558e70cda1f58374091ffac1153ff21bf8722eeb263"}
Oct 31 01:43:26.327937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.327899    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" event={"ID":"248a307a-a43d-4d2c-aade-59fb8e0efc2a","Type":"ContainerStarted","Data":"26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291"}
Oct 31 01:43:26.353650 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.353603    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" podStartSLOduration=2.353585916 podStartE2EDuration="2.353585916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:26.352355913 +0000 UTC m=+1364.301004306" watchObservedRunningTime="2026-10-31 01:43:26.353585916 +0000 UTC m=+1364.302234307"
Oct 31 01:43:26.536796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.536761    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:26.647694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.636500    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de343be-256a-4503-a8b4-55adc17989a0-kube-api-access-vjv2z\" (UniqueName: \"kubernetes.io/projected/0de343be-256a-4503-a8b4-55adc17989a0-kube-api-access-vjv2z\") pod \"0de343be-256a-4503-a8b4-55adc17989a0\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") "
Oct 31 01:43:26.647694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.636642    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-xks-ca\") pod \"0de343be-256a-4503-a8b4-55adc17989a0\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") "
Oct 31 01:43:26.647694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.636709    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-openshift-ca\") pod \"0de343be-256a-4503-a8b4-55adc17989a0\" (UID: \"0de343be-256a-4503-a8b4-55adc17989a0\") "
Oct 31 01:43:26.678062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.672841    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de343be-256a-4503-a8b4-55adc17989a0-kube-api-access-vjv2z" pod "0de343be-256a-4503-a8b4-55adc17989a0" (UID: "0de343be-256a-4503-a8b4-55adc17989a0"). InnerVolumeSpecName "kube-api-access-vjv2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:26.678062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.674344    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-xks-ca" pod "0de343be-256a-4503-a8b4-55adc17989a0" (UID: "0de343be-256a-4503-a8b4-55adc17989a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:26.678416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.678227    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-openshift-ca" pod "0de343be-256a-4503-a8b4-55adc17989a0" (UID: "0de343be-256a-4503-a8b4-55adc17989a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:26.738220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.738192    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:26.738220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.738218    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de343be-256a-4503-a8b4-55adc17989a0-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:26.738220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.738228    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjv2z\" (UniqueName: \"kubernetes.io/projected/0de343be-256a-4503-a8b4-55adc17989a0-kube-api-access-vjv2z\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:26.953617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.945055    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:26.964516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.964176    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:43:26.964516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:26.964239    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:43:27.337751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:27.337679    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"
Oct 31 01:43:27.338243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:27.337744    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc" event={"ID":"0de343be-256a-4503-a8b4-55adc17989a0","Type":"ContainerDied","Data":"2b5af7305093ab622ec4ceed65345dff610d046ff7602023c7568cd161cc4aba"}
Oct 31 01:43:27.338243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:27.337810    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b5af7305093ab622ec4ceed65345dff610d046ff7602023c7568cd161cc4aba"
Oct 31 01:43:27.939766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:27.939736    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:43:27.939943 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:27.939786    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:43:28.940064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:28.939443    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:43:28.940064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:28.939514    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:43:29.939420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:29.939378    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:43:29.939642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:29.939432    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:43:30.939718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:30.939689    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:43:30.940205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:30.939953    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:43:31.939700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:31.939656    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:43:31.939881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:31.939747    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:43:32.025229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.025195    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"]
Oct 31 01:43:32.025811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.025791    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0de343be-256a-4503-a8b4-55adc17989a0" containerName="revoke-keys"
Oct 31 01:43:32.025921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.025812    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de343be-256a-4503-a8b4-55adc17989a0" containerName="revoke-keys"
Oct 31 01:43:32.025981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.025959    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de343be-256a-4503-a8b4-55adc17989a0" containerName="revoke-keys"
Oct 31 01:43:32.029372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.029351    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.037469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.037356    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:32.037581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.037505    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:32.037679 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.037608    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:43:32.038656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.038631    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:32.048925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.048899    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"]
Oct 31 01:43:32.186617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.171800    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vhn\" (UniqueName: \"kubernetes.io/projected/6022b938-15c3-426e-8249-0aa49b38bfa0-kube-api-access-k6vhn\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.186617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.171965    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.186617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.172008    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.273107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.273014    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vhn\" (UniqueName: \"kubernetes.io/projected/6022b938-15c3-426e-8249-0aa49b38bfa0-kube-api-access-k6vhn\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.273519 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.273191    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.273654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.273569    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.273945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.273920    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.276164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.276139    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.287647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.287623    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vhn\" (UniqueName: \"kubernetes.io/projected/6022b938-15c3-426e-8249-0aa49b38bfa0-kube-api-access-k6vhn\") pod \"maas-api-revoke-sub-8f01db46-88gqj\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.348885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.348849    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:32.651271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.651095    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"]
Oct 31 01:43:32.655364 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:43:32.655337    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6022b938_15c3_426e_8249_0aa49b38bfa0.slice/crio-d1230ab8c36a3fc8229526e1a84a587169f7d828f550961d75bce17c2fa65a5e WatchSource:0}: Error finding container d1230ab8c36a3fc8229526e1a84a587169f7d828f550961d75bce17c2fa65a5e: Status 404 returned error can't find the container with id d1230ab8c36a3fc8229526e1a84a587169f7d828f550961d75bce17c2fa65a5e
Oct 31 01:43:32.945007 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.944978    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:32.945435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.945411    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:32.946125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:32.946096    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:43:33.378195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:33.378098    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj" event={"ID":"6022b938-15c3-426e-8249-0aa49b38bfa0","Type":"ContainerStarted","Data":"63df7a9e5c522872b7aada955ea9b2072d8248057710978c22044b7453c89eba"}
Oct 31 01:43:33.378195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:33.378168    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj" event={"ID":"6022b938-15c3-426e-8249-0aa49b38bfa0","Type":"ContainerStarted","Data":"d1230ab8c36a3fc8229526e1a84a587169f7d828f550961d75bce17c2fa65a5e"}
Oct 31 01:43:33.397528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:33.397471    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj" podStartSLOduration=1.397451478 podStartE2EDuration="1.397451478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:33.394158506 +0000 UTC m=+1371.342806899" watchObservedRunningTime="2026-10-31 01:43:33.397451478 +0000 UTC m=+1371.346099870"
Oct 31 01:43:33.632691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:33.632601    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:43:34.387805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:34.387767    5293 generic.go:356] "Generic (PLEG): container finished" podID="6022b938-15c3-426e-8249-0aa49b38bfa0" containerID="63df7a9e5c522872b7aada955ea9b2072d8248057710978c22044b7453c89eba" exitCode=0
Oct 31 01:43:34.388256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:34.387853    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj" event={"ID":"6022b938-15c3-426e-8249-0aa49b38bfa0","Type":"ContainerDied","Data":"63df7a9e5c522872b7aada955ea9b2072d8248057710978c22044b7453c89eba"}
Oct 31 01:43:35.813854 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:35.813823    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:35.941088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:35.941014    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-openshift-ca\") pod \"6022b938-15c3-426e-8249-0aa49b38bfa0\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") "
Oct 31 01:43:35.941255 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:35.941210    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6022b938-15c3-426e-8249-0aa49b38bfa0-kube-api-access-k6vhn\" (UniqueName: \"kubernetes.io/projected/6022b938-15c3-426e-8249-0aa49b38bfa0-kube-api-access-k6vhn\") pod \"6022b938-15c3-426e-8249-0aa49b38bfa0\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") "
Oct 31 01:43:35.941317 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:35.941301    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-xks-ca\") pod \"6022b938-15c3-426e-8249-0aa49b38bfa0\" (UID: \"6022b938-15c3-426e-8249-0aa49b38bfa0\") "
Oct 31 01:43:35.941379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:35.941358    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-openshift-ca" pod "6022b938-15c3-426e-8249-0aa49b38bfa0" (UID: "6022b938-15c3-426e-8249-0aa49b38bfa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:35.941830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:35.941808    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:35.943518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:35.943494    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-xks-ca" pod "6022b938-15c3-426e-8249-0aa49b38bfa0" (UID: "6022b938-15c3-426e-8249-0aa49b38bfa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:35.943640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:35.943559    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6022b938-15c3-426e-8249-0aa49b38bfa0-kube-api-access-k6vhn" pod "6022b938-15c3-426e-8249-0aa49b38bfa0" (UID: "6022b938-15c3-426e-8249-0aa49b38bfa0"). InnerVolumeSpecName "kube-api-access-k6vhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:36.045101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:36.044323    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vhn\" (UniqueName: \"kubernetes.io/projected/6022b938-15c3-426e-8249-0aa49b38bfa0-kube-api-access-k6vhn\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:36.045101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:36.044360    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6022b938-15c3-426e-8249-0aa49b38bfa0-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:36.418148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:36.414101    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj" event={"ID":"6022b938-15c3-426e-8249-0aa49b38bfa0","Type":"ContainerDied","Data":"d1230ab8c36a3fc8229526e1a84a587169f7d828f550961d75bce17c2fa65a5e"}
Oct 31 01:43:36.418148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:36.414168    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1230ab8c36a3fc8229526e1a84a587169f7d828f550961d75bce17c2fa65a5e"
Oct 31 01:43:36.418148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:36.414236    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"
Oct 31 01:43:38.209849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.207367    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"]
Oct 31 01:43:38.213934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.213908    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6022b938-15c3-426e-8249-0aa49b38bfa0" containerName="revoke-keys"
Oct 31 01:43:38.213934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.213931    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6022b938-15c3-426e-8249-0aa49b38bfa0" containerName="revoke-keys"
Oct 31 01:43:38.214220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.214071    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6022b938-15c3-426e-8249-0aa49b38bfa0" containerName="revoke-keys"
Oct 31 01:43:38.218104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.218084    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.243338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.242705    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"]
Oct 31 01:43:38.285224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285162    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.285224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285215    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.285550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285313    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fn8\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-kube-api-access-82fn8\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.285550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285449    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.285550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285484    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.285550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285530    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.285789 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285672    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.285789 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285708    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225a7a3-dddd-41c3-b5fd-b6806209befd-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.285789 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.285742    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.287553 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.287528    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"]
Oct 31 01:43:38.291453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.291433    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.294394 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.294375    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"distinct-w4-be6935-kserve-self-signed-certs\""
Oct 31 01:43:38.303503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.303479    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"]
Oct 31 01:43:38.395326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.390276    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h82b\" (UniqueName: \"kubernetes.io/projected/2ef71ae1-dac7-470a-941d-e3c2f83f6940-kube-api-access-6h82b\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.395650 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.395623    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.395778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.395764    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225a7a3-dddd-41c3-b5fd-b6806209befd-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.398154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.398124    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.402221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.402190    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.402408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.402390    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.402514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.402502    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tls-certs\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.402613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.402600    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fn8\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-kube-api-access-82fn8\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.402790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.402775    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-home\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.402900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.402884    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-dshm\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.408781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.408754    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.408973 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.408956    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.409163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.409145    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.409338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.409318    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tmp-dir\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.409454 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.409439    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-model-cache\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.421119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.410806    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225a7a3-dddd-41c3-b5fd-b6806209befd-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.421331 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.417725    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.421428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.417973    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.421801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.421776    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.423824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.423800    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.485588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.484680    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.485588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.484840    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fn8\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-kube-api-access-82fn8\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.488374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.486788    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.504106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.504070    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.510732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.510705    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tmp-dir\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.510838 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.510744    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-model-cache\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.510838 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.510812    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h82b\" (UniqueName: \"kubernetes.io/projected/2ef71ae1-dac7-470a-941d-e3c2f83f6940-kube-api-access-6h82b\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.510907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.510895    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tls-certs\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.511000 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.510983    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-home\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.511125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.511013    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-dshm\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.511783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.511759    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tmp-dir\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.512020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.511995    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-model-cache\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.512723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.512018    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-home\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.513919 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.513890    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-dshm\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.514735 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.514718    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tls-certs\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.523099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.523074    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h82b\" (UniqueName: \"kubernetes.io/projected/2ef71ae1-dac7-470a-941d-e3c2f83f6940-kube-api-access-6h82b\") pod \"distinct-w4-be6935-kserve-8c7f9b865-rn7qf\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.545554 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.545532    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:38.605817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.605632    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:38.835491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.835466    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"]
Oct 31 01:43:38.837786 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:43:38.837760    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf225a7a3_dddd_41c3_b5fd_b6806209befd.slice/crio-3555f0068a0f415cc06e54ce3baf00d25a10aeda487f512271cb360d91b614af WatchSource:0}: Error finding container 3555f0068a0f415cc06e54ce3baf00d25a10aeda487f512271cb360d91b614af: Status 404 returned error can't find the container with id 3555f0068a0f415cc06e54ce3baf00d25a10aeda487f512271cb360d91b614af
Oct 31 01:43:38.840172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.840113    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:38.840263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.840180    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:38.840263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.840211    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:38.916818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:38.916793    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"]
Oct 31 01:43:38.918705 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:43:38.918680    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ef71ae1_dac7_470a_941d_e3c2f83f6940.slice/crio-c72c95c450566932c121db1261275e7dd6ca4d502664334e8d52a329db8c442a WatchSource:0}: Error finding container c72c95c450566932c121db1261275e7dd6ca4d502664334e8d52a329db8c442a: Status 404 returned error can't find the container with id c72c95c450566932c121db1261275e7dd6ca4d502664334e8d52a329db8c442a
Oct 31 01:43:39.560449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.560410    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" event={"ID":"f225a7a3-dddd-41c3-b5fd-b6806209befd","Type":"ContainerStarted","Data":"06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054"}
Oct 31 01:43:39.561127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.561100    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" event={"ID":"f225a7a3-dddd-41c3-b5fd-b6806209befd","Type":"ContainerStarted","Data":"3555f0068a0f415cc06e54ce3baf00d25a10aeda487f512271cb360d91b614af"}
Oct 31 01:43:39.588079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.588029    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:39.589916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.589891    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:43:39.590193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.590128    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:43:39.627086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.622739    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podStartSLOduration=1.622720377 podStartE2EDuration="1.622720377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:39.621286431 +0000 UTC m=+1377.569934824" watchObservedRunningTime="2026-10-31 01:43:39.622720377 +0000 UTC m=+1377.571368769"
Oct 31 01:43:39.635088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.632752    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" event={"ID":"2ef71ae1-dac7-470a-941d-e3c2f83f6940","Type":"ContainerStarted","Data":"a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb"}
Oct 31 01:43:39.635088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.632800    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" event={"ID":"2ef71ae1-dac7-470a-941d-e3c2f83f6940","Type":"ContainerStarted","Data":"c72c95c450566932c121db1261275e7dd6ca4d502664334e8d52a329db8c442a"}
Oct 31 01:43:39.653461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:39.653413    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podStartSLOduration=1.6533982809999999 podStartE2EDuration="1.653398281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:39.651244943 +0000 UTC m=+1377.599893335" watchObservedRunningTime="2026-10-31 01:43:39.653398281 +0000 UTC m=+1377.602046673"
Oct 31 01:43:40.546156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:40.546118    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:43:40.546321 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:40.546178    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:43:40.611153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:40.611118    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:41.549430 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:41.549391    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:43:41.549589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:41.549454    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:43:41.779574 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:43:41.779531    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:43:42.553764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:42.553698    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:43:42.553975 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:42.553767    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:43:43.546867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:43.546544    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:43:43.546867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:43.546589    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:43:44.546270 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.546235    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:43:44.546521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.546489    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:43:44.729815 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.729769    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"]
Oct 31 01:43:44.743826 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.743772    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.746585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.746528    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"]
Oct 31 01:43:44.747696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.747673    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:44.747819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.747766    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:44.751066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.751024    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:43:44.843998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.843890    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.843998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.843940    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw9gg\" (UniqueName: \"kubernetes.io/projected/13aa1f63-03a8-425b-8385-9177f86d95b7-kube-api-access-lw9gg\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.844280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.844027    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.847731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.847687    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:44.945102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.945060    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.945313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.945122    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9gg\" (UniqueName: \"kubernetes.io/projected/13aa1f63-03a8-425b-8385-9177f86d95b7-kube-api-access-lw9gg\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.945313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.945264    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.945727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.945707    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.947788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.947761    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:44.956443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:44.956384    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw9gg\" (UniqueName: \"kubernetes.io/projected/13aa1f63-03a8-425b-8385-9177f86d95b7-kube-api-access-lw9gg\") pod \"maas-api-revoke-sub-8703b524-bhhpr\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:45.062323 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:45.062280    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:45.356238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:45.356200    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"]
Oct 31 01:43:45.357001 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:43:45.356970    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13aa1f63_03a8_425b_8385_9177f86d95b7.slice/crio-15a303478b9498f1d7e9a33ea926345d8caaa57383d622af5fb76f8d7aa53b18 WatchSource:0}: Error finding container 15a303478b9498f1d7e9a33ea926345d8caaa57383d622af5fb76f8d7aa53b18: Status 404 returned error can't find the container with id 15a303478b9498f1d7e9a33ea926345d8caaa57383d622af5fb76f8d7aa53b18
Oct 31 01:43:45.545957 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:45.545924    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:43:45.546134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:45.545986    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:43:45.677665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:45.677629    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr" event={"ID":"13aa1f63-03a8-425b-8385-9177f86d95b7","Type":"ContainerStarted","Data":"18d4e5c8b9a592436ff597465f526b2debb63df9716ecffafcc095c0dc5df23a"}
Oct 31 01:43:45.677810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:45.677690    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr" event={"ID":"13aa1f63-03a8-425b-8385-9177f86d95b7","Type":"ContainerStarted","Data":"15a303478b9498f1d7e9a33ea926345d8caaa57383d622af5fb76f8d7aa53b18"}
Oct 31 01:43:45.696133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:45.696084    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr" podStartSLOduration=1.696066562 podStartE2EDuration="1.696066562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:45.6936754 +0000 UTC m=+1383.642323791" watchObservedRunningTime="2026-10-31 01:43:45.696066562 +0000 UTC m=+1383.644714962"
Oct 31 01:43:46.732071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:46.731204    5293 generic.go:356] "Generic (PLEG): container finished" podID="13aa1f63-03a8-425b-8385-9177f86d95b7" containerID="18d4e5c8b9a592436ff597465f526b2debb63df9716ecffafcc095c0dc5df23a" exitCode=0
Oct 31 01:43:46.732071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:46.731442    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr" event={"ID":"13aa1f63-03a8-425b-8385-9177f86d95b7","Type":"ContainerDied","Data":"18d4e5c8b9a592436ff597465f526b2debb63df9716ecffafcc095c0dc5df23a"}
Oct 31 01:43:47.384067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:47.383608    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:47.554897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:47.554861    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:47.555174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:47.555141    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:47.613165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:47.613120    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:47.659172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:47.659129    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.043528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.043423    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.285055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.284992    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.396719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.396678    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.426787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.423110    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.426787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.423544    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:43:48.566072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.563151    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.644608 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.644566    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.647961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.646301    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:48.692073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.647431    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:43:48.889067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.754713    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:43:48.983390 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.770292    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:48.985024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.984989    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.985274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:48.814103    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.320121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.316795    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.335278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.334252    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:43:49.417696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.397587    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:43:49.567067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.531990    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.567067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.532683    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.567067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.534075    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:43:49.567067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.541434    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:43:49.567067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.541535    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:49.567067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.541597    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:43:49.624618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.603581    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.624618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.614232    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:43:49.644200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.644138    5293 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.688194 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.688142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:43:50.013860 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.803588    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:43:50.013860 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.837326    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:50.013860 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.907847    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.013860 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.936135    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.044146 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:49.909022    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13aa1f63-03a8-425b-8385-9177f86d95b7-kube-api-access-lw9gg\" (UniqueName: \"kubernetes.io/projected/13aa1f63-03a8-425b-8385-9177f86d95b7-kube-api-access-lw9gg\") pod \"13aa1f63-03a8-425b-8385-9177f86d95b7\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") "
Oct 31 01:43:50.044342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.044295    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-xks-ca\") pod \"13aa1f63-03a8-425b-8385-9177f86d95b7\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") "
Oct 31 01:43:50.044416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.044376    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-openshift-ca\") pod \"13aa1f63-03a8-425b-8385-9177f86d95b7\" (UID: \"13aa1f63-03a8-425b-8385-9177f86d95b7\") "
Oct 31 01:43:50.099173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.094287    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-openshift-ca" pod "13aa1f63-03a8-425b-8385-9177f86d95b7" (UID: "13aa1f63-03a8-425b-8385-9177f86d95b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:50.170090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.162598    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:50.214337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.214280    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-xks-ca" pod "13aa1f63-03a8-425b-8385-9177f86d95b7" (UID: "13aa1f63-03a8-425b-8385-9177f86d95b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:50.270339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.270230    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13aa1f63-03a8-425b-8385-9177f86d95b7-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:50.454067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.441764    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13aa1f63-03a8-425b-8385-9177f86d95b7-kube-api-access-lw9gg" pod "13aa1f63-03a8-425b-8385-9177f86d95b7" (UID: "13aa1f63-03a8-425b-8385-9177f86d95b7"). InnerVolumeSpecName "kube-api-access-lw9gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:50.546788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.546191    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.577337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.577281    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.577726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.577590    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:43:50.578219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.577634    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:43:50.591218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.591176    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.593286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.593242    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw9gg\" (UniqueName: \"kubernetes.io/projected/13aa1f63-03a8-425b-8385-9177f86d95b7-kube-api-access-lw9gg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:50.608646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.608598    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.619834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.619800    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr" event={"ID":"13aa1f63-03a8-425b-8385-9177f86d95b7","Type":"ContainerDied","Data":"15a303478b9498f1d7e9a33ea926345d8caaa57383d622af5fb76f8d7aa53b18"}
Oct 31 01:43:50.620065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.620027    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15a303478b9498f1d7e9a33ea926345d8caaa57383d622af5fb76f8d7aa53b18"
Oct 31 01:43:50.620261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.620247    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"
Oct 31 01:43:50.731012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.730544    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:50.731012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.730606    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.758186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.758135    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.913996 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.913949    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:51.065612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.939115    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:43:51.066836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:50.989604    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:43:51.067103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.067083    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:43:51.067271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.067257    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:43:51.102301 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.102275    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:43:51.219817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.219740    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:43:51.219977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.219824    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:51.219977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.219873    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:43:51.223828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.223804    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:51.226297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.226275    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 01:43:51.248911 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.248881    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:43:51.277573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:51.277542    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:43:55.934102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:55.933415    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:43:56.616076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.612137    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"]
Oct 31 01:43:56.616076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.613008    5293 cpu_manager.go:414] "containerMap: removing container" podUID="13aa1f63-03a8-425b-8385-9177f86d95b7" containerName="revoke-keys"
Oct 31 01:43:56.616076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.613030    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13aa1f63-03a8-425b-8385-9177f86d95b7" containerName="revoke-keys"
Oct 31 01:43:56.616076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.613249    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="13aa1f63-03a8-425b-8385-9177f86d95b7" containerName="revoke-keys"
Oct 31 01:43:56.618441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.618417    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"]
Oct 31 01:43:56.618706 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.618693    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.639575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.632972    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"]
Oct 31 01:43:56.639575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.636152    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:56.639575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.636311    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:56.639575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.637462    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:43:56.639575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.637545    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:56.669425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.669390    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.671487 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.671460    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"]
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693429    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693495    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693547    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68dlv\" (UniqueName: \"kubernetes.io/projected/db94e412-6e99-4bc8-bf33-2b258cfbaeba-kube-api-access-68dlv\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693572    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c684b0-2883-49c6-b162-bab1e566f591-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693613    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693687    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693721    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c684b0-2883-49c6-b162-bab1e566f591-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693766    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n77gz\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-kube-api-access-n77gz\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.693812    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.694115    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.694162    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.695099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.695020    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796220    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.796492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796298    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796349    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68dlv\" (UniqueName: \"kubernetes.io/projected/db94e412-6e99-4bc8-bf33-2b258cfbaeba-kube-api-access-68dlv\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.796492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796371    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c684b0-2883-49c6-b162-bab1e566f591-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796413    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796482    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.796798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796515    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c684b0-2883-49c6-b162-bab1e566f591-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796546    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n77gz\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-kube-api-access-n77gz\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796587    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796652    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796678    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.796798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.796706    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.797939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.797916    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.798389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.798098    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.799182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.798590    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.799875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.799832    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c684b0-2883-49c6-b162-bab1e566f591-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.800695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.800671    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.801481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.801462    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.807973 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.807920    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c684b0-2883-49c6-b162-bab1e566f591-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.808493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.808468    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.808597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.808469    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:56.812500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.812433    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.813239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.813190    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n77gz\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-kube-api-access-n77gz\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:56.813862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:56.813836    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68dlv\" (UniqueName: \"kubernetes.io/projected/db94e412-6e99-4bc8-bf33-2b258cfbaeba-kube-api-access-68dlv\") pod \"maas-api-revoke-sub-35be93c2-92s44\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:57.010827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.010535    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:43:57.038094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.033301    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:43:57.108374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.081555    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"]
Oct 31 01:43:57.108374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.086679    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.124219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.122964    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.124219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.123193    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zvl\" (UniqueName: \"kubernetes.io/projected/464b1fc8-03b2-4d95-aece-0e33f134b266-kube-api-access-j7zvl\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.124219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.123283    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.140090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.127091    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"]
Oct 31 01:43:57.302132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.226943    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zvl\" (UniqueName: \"kubernetes.io/projected/464b1fc8-03b2-4d95-aece-0e33f134b266-kube-api-access-j7zvl\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.302132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.227029    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.302132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.245554    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.302132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.246654    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.302132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.277108    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zvl\" (UniqueName: \"kubernetes.io/projected/464b1fc8-03b2-4d95-aece-0e33f134b266-kube-api-access-j7zvl\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.302132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.277552    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac46e55-97rlv\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:57.591117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:57.560636    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:43:58.144387 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.029379    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"]
Oct 31 01:43:58.154617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.150991    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:58.154617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.151170    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:58.154617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.151219    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:43:58.489580 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.489459    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"]
Oct 31 01:43:58.617250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.617178    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:58.992316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.934324    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:58.992316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.945965    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:58.992316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:58.991429    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:59.012851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:59.010745    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:43:59.013500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:59.013225    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:59.082513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:59.081236    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" event={"ID":"34c684b0-2883-49c6-b162-bab1e566f591","Type":"ContainerStarted","Data":"2c67b50fbbfd5fb494f72528fbefc3e53afb0430b12c67f8dc142702094744bc"}
Oct 31 01:43:59.082931 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:59.082869    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44" event={"ID":"db94e412-6e99-4bc8-bf33-2b258cfbaeba","Type":"ContainerStarted","Data":"f3872ef9ec65ee99086af37c37163c40f29d5cffba0656c7ee89899001116a01"}
Oct 31 01:43:59.138453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:43:59.138406    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"]
Oct 31 01:43:59.158667 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:43:59.158640    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464b1fc8_03b2_4d95_aece_0e33f134b266.slice/crio-1b40e3571ec7fc33df5039a5449ed28425f720a68a9d2774d625e4747a521482 WatchSource:0}: Error finding container 1b40e3571ec7fc33df5039a5449ed28425f720a68a9d2774d625e4747a521482: Status 404 returned error can't find the container with id 1b40e3571ec7fc33df5039a5449ed28425f720a68a9d2774d625e4747a521482
Oct 31 01:44:00.091835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:00.091219    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv" event={"ID":"464b1fc8-03b2-4d95-aece-0e33f134b266","Type":"ContainerStarted","Data":"98c48f172cc3f884cac8bab05374a3f5f625d830bed053e6175aebe6b8b9a078"}
Oct 31 01:44:00.091835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:00.091281    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv" event={"ID":"464b1fc8-03b2-4d95-aece-0e33f134b266","Type":"ContainerStarted","Data":"1b40e3571ec7fc33df5039a5449ed28425f720a68a9d2774d625e4747a521482"}
Oct 31 01:44:00.093182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:00.093145    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" event={"ID":"34c684b0-2883-49c6-b162-bab1e566f591","Type":"ContainerStarted","Data":"f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5"}
Oct 31 01:44:00.126150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:00.125348    5293 generic.go:356] "Generic (PLEG): container finished" podID="db94e412-6e99-4bc8-bf33-2b258cfbaeba" containerID="9c10d58fdb576f1354bda893a0726f1098145f367faf5b9377f87937232fcf24" exitCode=0
Oct 31 01:44:00.126150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:00.125427    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv" podStartSLOduration=3.125413011 podStartE2EDuration="3.125413011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:00.113610884 +0000 UTC m=+1398.062259279" watchObservedRunningTime="2026-10-31 01:44:00.125413011 +0000 UTC m=+1398.074061406"
Oct 31 01:44:00.126150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:00.125481    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44" event={"ID":"db94e412-6e99-4bc8-bf33-2b258cfbaeba","Type":"ContainerDied","Data":"9c10d58fdb576f1354bda893a0726f1098145f367faf5b9377f87937232fcf24"}
Oct 31 01:44:00.145349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:00.145299    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podStartSLOduration=4.145279289 podStartE2EDuration="4.145279289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:00.144072309 +0000 UTC m=+1398.092720701" watchObservedRunningTime="2026-10-31 01:44:00.145279289 +0000 UTC m=+1398.093927724"
Oct 31 01:44:01.039501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.039463    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:44:01.041186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.041154    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:44:01.041286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.041218    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:44:01.135446 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.135413    5293 generic.go:356] "Generic (PLEG): container finished" podID="464b1fc8-03b2-4d95-aece-0e33f134b266" containerID="98c48f172cc3f884cac8bab05374a3f5f625d830bed053e6175aebe6b8b9a078" exitCode=0
Oct 31 01:44:01.135641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.135494    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv" event={"ID":"464b1fc8-03b2-4d95-aece-0e33f134b266","Type":"ContainerDied","Data":"98c48f172cc3f884cac8bab05374a3f5f625d830bed053e6175aebe6b8b9a078"}
Oct 31 01:44:01.374442 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.374413    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:44:01.472069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.466591    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-openshift-ca\") pod \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") "
Oct 31 01:44:01.472069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.466646    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db94e412-6e99-4bc8-bf33-2b258cfbaeba-kube-api-access-68dlv\" (UniqueName: \"kubernetes.io/projected/db94e412-6e99-4bc8-bf33-2b258cfbaeba-kube-api-access-68dlv\") pod \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") "
Oct 31 01:44:01.472069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.466708    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-xks-ca\") pod \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\" (UID: \"db94e412-6e99-4bc8-bf33-2b258cfbaeba\") "
Oct 31 01:44:01.519112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.517502    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-openshift-ca" pod "db94e412-6e99-4bc8-bf33-2b258cfbaeba" (UID: "db94e412-6e99-4bc8-bf33-2b258cfbaeba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:01.544094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.535532    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:01.544094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.535691    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-xks-ca" pod "db94e412-6e99-4bc8-bf33-2b258cfbaeba" (UID: "db94e412-6e99-4bc8-bf33-2b258cfbaeba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:01.557090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.556241    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db94e412-6e99-4bc8-bf33-2b258cfbaeba-kube-api-access-68dlv" pod "db94e412-6e99-4bc8-bf33-2b258cfbaeba" (UID: "db94e412-6e99-4bc8-bf33-2b258cfbaeba"). InnerVolumeSpecName "kube-api-access-68dlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:01.636928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.636849    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68dlv\" (UniqueName: \"kubernetes.io/projected/db94e412-6e99-4bc8-bf33-2b258cfbaeba-kube-api-access-68dlv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:01.636928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:01.636876    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db94e412-6e99-4bc8-bf33-2b258cfbaeba-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:02.039340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.039306    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:44:02.039514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.039366    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:44:02.149580 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.149554    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"
Oct 31 01:44:02.149982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.149595    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44" event={"ID":"db94e412-6e99-4bc8-bf33-2b258cfbaeba","Type":"ContainerDied","Data":"f3872ef9ec65ee99086af37c37163c40f29d5cffba0656c7ee89899001116a01"}
Oct 31 01:44:02.149982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.149672    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3872ef9ec65ee99086af37c37163c40f29d5cffba0656c7ee89899001116a01"
Oct 31 01:44:02.444408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.444372    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:44:02.570916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.570883    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-openshift-ca\") pod \"464b1fc8-03b2-4d95-aece-0e33f134b266\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") "
Oct 31 01:44:02.571141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.570952    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-xks-ca\") pod \"464b1fc8-03b2-4d95-aece-0e33f134b266\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") "
Oct 31 01:44:02.571141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.571020    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/464b1fc8-03b2-4d95-aece-0e33f134b266-kube-api-access-j7zvl\" (UniqueName: \"kubernetes.io/projected/464b1fc8-03b2-4d95-aece-0e33f134b266-kube-api-access-j7zvl\") pod \"464b1fc8-03b2-4d95-aece-0e33f134b266\" (UID: \"464b1fc8-03b2-4d95-aece-0e33f134b266\") "
Oct 31 01:44:02.571289 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.571262    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-openshift-ca" pod "464b1fc8-03b2-4d95-aece-0e33f134b266" (UID: "464b1fc8-03b2-4d95-aece-0e33f134b266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:02.571583 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.571564    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:02.573223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.573195    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-xks-ca" pod "464b1fc8-03b2-4d95-aece-0e33f134b266" (UID: "464b1fc8-03b2-4d95-aece-0e33f134b266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:02.573349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.573334    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464b1fc8-03b2-4d95-aece-0e33f134b266-kube-api-access-j7zvl" pod "464b1fc8-03b2-4d95-aece-0e33f134b266" (UID: "464b1fc8-03b2-4d95-aece-0e33f134b266"). InnerVolumeSpecName "kube-api-access-j7zvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:02.672225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.672195    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/464b1fc8-03b2-4d95-aece-0e33f134b266-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:02.672225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:02.672219    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7zvl\" (UniqueName: \"kubernetes.io/projected/464b1fc8-03b2-4d95-aece-0e33f134b266-kube-api-access-j7zvl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:03.039751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:03.039657    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:44:03.039751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:03.039729    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:44:03.199074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:03.198106    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv" event={"ID":"464b1fc8-03b2-4d95-aece-0e33f134b266","Type":"ContainerDied","Data":"1b40e3571ec7fc33df5039a5449ed28425f720a68a9d2774d625e4747a521482"}
Oct 31 01:44:03.199074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:03.198173    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b40e3571ec7fc33df5039a5449ed28425f720a68a9d2774d625e4747a521482"
Oct 31 01:44:03.199074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:03.198247    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"
Oct 31 01:44:04.038891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.038853    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:44:04.039100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.038912    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:44:04.827227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.827191    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"]
Oct 31 01:44:04.828571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.828544    5293 cpu_manager.go:414] "containerMap: removing container" podUID="464b1fc8-03b2-4d95-aece-0e33f134b266" containerName="revoke-keys"
Oct 31 01:44:04.828760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.828746    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="464b1fc8-03b2-4d95-aece-0e33f134b266" containerName="revoke-keys"
Oct 31 01:44:04.828896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.828885    5293 cpu_manager.go:414] "containerMap: removing container" podUID="db94e412-6e99-4bc8-bf33-2b258cfbaeba" containerName="revoke-keys"
Oct 31 01:44:04.828970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.828961    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db94e412-6e99-4bc8-bf33-2b258cfbaeba" containerName="revoke-keys"
Oct 31 01:44:04.829216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.829202    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="db94e412-6e99-4bc8-bf33-2b258cfbaeba" containerName="revoke-keys"
Oct 31 01:44:04.829328 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.829318    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="464b1fc8-03b2-4d95-aece-0e33f134b266" containerName="revoke-keys"
Oct 31 01:44:04.834162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.834126    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.838231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.838203    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"]
Oct 31 01:44:04.852620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.852597    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"]
Oct 31 01:44:04.852851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.852835    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.872281 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.872245    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"]
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922666    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922719    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922751    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47glx\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-kube-api-access-47glx\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922785    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922814    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922846    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922870    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4l6\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-kube-api-access-gz4l6\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922904    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.922958    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aeddbd-9e95-437d-b476-c5fc291339b0-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923026    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923076    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923101    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923156    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644859-8273-4219-b427-970e6e36d505-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923199    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923244    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644859-8273-4219-b427-970e6e36d505-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923289    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923330    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:04.928100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:04.923361    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.026769    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.026828    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4l6\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-kube-api-access-gz4l6\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.026865    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.027063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.026920    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aeddbd-9e95-437d-b476-c5fc291339b0-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.027063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.026994    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.027063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027024    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027070    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027561    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027648    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644859-8273-4219-b427-970e6e36d505-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027706    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027711    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.027807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027784    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644859-8273-4219-b427-970e6e36d505-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.027968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027829    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.027968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027882    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.027968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.027910    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.028163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.028005    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.028163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.028079    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.028163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.028110    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47glx\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-kube-api-access-47glx\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.028163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.028147    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.028458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.028439    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.028678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.028659    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.028874 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.028852    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aeddbd-9e95-437d-b476-c5fc291339b0-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.028968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.028906    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.029867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.029805    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.030693 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.030666    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.031265 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.031239    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.031817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.031786    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.032461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.032438    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644859-8273-4219-b427-970e6e36d505-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.035625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.035604    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644859-8273-4219-b427-970e6e36d505-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.035960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.035941    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.037928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.037908    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.038797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.038480    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.038882 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.038841    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:44:05.038938 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.038880    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:44:05.041394 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.041374    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.042868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.042826    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4l6\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-kube-api-access-gz4l6\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.043845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.043826    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47glx\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-kube-api-access-47glx\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.044535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.044515    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.166085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.165714    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:05.172224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.172184    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:05.479228 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.479194    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"]
Oct 31 01:44:05.483344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.483305    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:05.483454 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.483386    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:05.483454 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.483425    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:05.585160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.585139    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"]
Oct 31 01:44:05.587477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.587435    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:05.587559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.587523    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:05.587612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:05.587564    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:06.041556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.041115    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:44:06.041556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.041174    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:44:06.378990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.378891    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" event={"ID":"55aeddbd-9e95-437d-b476-c5fc291339b0","Type":"ContainerStarted","Data":"8232cc6607897a904f877232a4792cb6d90322625f4fa4537fdb27b9f920499f"}
Oct 31 01:44:06.378990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.378979    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" event={"ID":"55aeddbd-9e95-437d-b476-c5fc291339b0","Type":"ContainerStarted","Data":"351667a55fbf4dd89c8f004a4e1303b9e94e91f51d347e4b5a9cf75c932a47b8"}
Oct 31 01:44:06.380660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.380637    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" event={"ID":"67644859-8273-4219-b427-970e6e36d505","Type":"ContainerStarted","Data":"3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5"}
Oct 31 01:44:06.380848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.380835    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" event={"ID":"67644859-8273-4219-b427-970e6e36d505","Type":"ContainerStarted","Data":"25639d26779eb61ca583b6aff353203bc94e39f8c226a1b44738bca32c1162a3"}
Oct 31 01:44:06.441342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.441276    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podStartSLOduration=2.441251279 podStartE2EDuration="2.441251279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:06.402674423 +0000 UTC m=+1404.351322824" watchObservedRunningTime="2026-10-31 01:44:06.441251279 +0000 UTC m=+1404.389899672"
Oct 31 01:44:06.466067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.444660    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podStartSLOduration=2.444641365 podStartE2EDuration="2.444641365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:06.430644781 +0000 UTC m=+1404.379293178" watchObservedRunningTime="2026-10-31 01:44:06.444641365 +0000 UTC m=+1404.393289760"
Oct 31 01:44:06.870177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.870107    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"]
Oct 31 01:44:06.895093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.892467    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"]
Oct 31 01:44:06.895093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.892597    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:06.920076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.906348    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:44:06.920076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.906578    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:06.920076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.906630    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:06.920076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.907167    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:44:06.994875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.994840    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvk9v\" (UniqueName: \"kubernetes.io/projected/5003abae-3098-4227-8b03-49f37e126502-kube-api-access-kvk9v\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:06.995264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.995226    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5003abae-3098-4227-8b03-49f37e126502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:06.995496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:06.995478    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5003abae-3098-4227-8b03-49f37e126502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:07.127119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.113918    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5003abae-3098-4227-8b03-49f37e126502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:07.149459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.149426    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5003abae-3098-4227-8b03-49f37e126502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:07.149931 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.149912    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvk9v\" (UniqueName: \"kubernetes.io/projected/5003abae-3098-4227-8b03-49f37e126502-kube-api-access-kvk9v\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:07.161235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.161010    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5003abae-3098-4227-8b03-49f37e126502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:07.162218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.162195    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5003abae-3098-4227-8b03-49f37e126502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:07.175875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.175852    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:07.183297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.183274    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:07.238362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.191294    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 01:44:07.238687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.238642    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 01:44:07.280163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.255873    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:07.280163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.255946    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:44:07.280163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.258476    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvk9v\" (UniqueName: \"kubernetes.io/projected/5003abae-3098-4227-8b03-49f37e126502-kube-api-access-kvk9v\") pod \"maas-api-revoke-sub-d8d16e32-qwk89\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:07.281067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.280997    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:07.366094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.361391    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:44:07.379103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.367707    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:44:07.385150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:07.384382    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:44:08.181001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.180964    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 01:44:08.181625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.181592    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 01:44:08.236343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.236309    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:08.237593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.237561    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:44:08.514895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.514850    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"]
Oct 31 01:44:08.544066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.538638    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.677089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.674599    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdt2\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-kube-api-access-lxdt2\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.677089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.674662    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.677089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.674772    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.677089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.674915    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.677089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.675019    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.722105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.719584    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:08.772196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.771008    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.772196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.771106    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.772196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.771261    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.772196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.771358    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.780244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.774487    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:08.818078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.807676    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"]
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873522    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdt2\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-kube-api-access-lxdt2\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873581    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873638    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873699    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873759    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873794    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873831    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873953    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.873992    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.876483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.874589    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.888088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.886374    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.888088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.886666    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.888088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.887741    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.900064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.896378    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.920539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.905465    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.920539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.910643    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:08.962455 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.962339    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"]
Oct 31 01:44:08.964518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:08.964419    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdt2\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-kube-api-access-lxdt2\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:09.022661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.022107    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:09.024059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.022897    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:44:09.029622 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.029597    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:09.115667 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.112174    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:09.115667 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.114788    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:09.292129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.290818    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 01:44:09.292129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.290888    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 01:44:09.311739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.294011    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:09.311739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.299957    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:09.311739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.300016    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:44:09.512066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.511676    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:09.709199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.684093    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"]
Oct 31 01:44:09.709199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.687603    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:09.726500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.723227    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.726500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.723970    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:09.726500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.724516    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:44:09.730831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.730807    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"]
Oct 31 01:44:09.756075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.755973    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89" event={"ID":"5003abae-3098-4227-8b03-49f37e126502","Type":"ContainerStarted","Data":"d9be322068ae4d51b9c673902d6924a3644ddedcaebfa3d0c5e39e6b5a8f5e1f"}
Oct 31 01:44:09.786314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.786261    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:09.794082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.787465    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:44:09.872009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.871917    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.876237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.876205    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.876546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.876527    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.876724 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.876706    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.876937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.876919    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f54a9a0-1e40-4213-a73e-554d493f9313-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.877165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.877145    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.877376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.877359    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99l6\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-kube-api-access-c99l6\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.877568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.877537    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.877841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.877824    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.958273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.958211    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981005    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981082    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99l6\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-kube-api-access-c99l6\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981128    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981235    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981332    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981367    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981430    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981468    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.981519    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f54a9a0-1e40-4213-a73e-554d493f9313-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.982662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.982310    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f54a9a0-1e40-4213-a73e-554d493f9313-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.984303 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.983643    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.985027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.984980    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.985514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.985495    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:09.985948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:09.985912    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:10.070065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.062121    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99l6\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-kube-api-access-c99l6\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:10.102075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.087165    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:10.148121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.146350    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:10.302063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.299720    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:10.302063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.299779    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:44:10.412792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.327644    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:10.412792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.328372    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 01:44:10.412792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.328428    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 01:44:10.412792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.328752    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:10.548066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.547166    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:10.548066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.547582    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:44:10.592385 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.592344    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:10.695381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.695347    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"]
Oct 31 01:44:10.747538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.747495    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:10.767741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.767198    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:10.789571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.789241    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:10.883661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.876849    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:44:10.883661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.880348    5293 generic.go:356] "Generic (PLEG): container finished" podID="5003abae-3098-4227-8b03-49f37e126502" containerID="3aa82a556f25598da65c83b972dd70a42396c5daf46d18416f6cae0768e3ed4c" exitCode=0
Oct 31 01:44:10.883661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.881258    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89" event={"ID":"5003abae-3098-4227-8b03-49f37e126502","Type":"ContainerDied","Data":"3aa82a556f25598da65c83b972dd70a42396c5daf46d18416f6cae0768e3ed4c"}
Oct 31 01:44:10.883661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.882208    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:10.883661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.882281    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:10.883661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.882329    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:10.887400 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.886755    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:44:10.888586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:10.888564    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:44:11.160891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.160341    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"]
Oct 31 01:44:11.168786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.168751    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:11.168969 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.168805    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:44:11.170712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.170683    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"]
Oct 31 01:44:11.171020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.170999    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.174076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.173604    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 01:44:11.174076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.173666    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 01:44:11.185119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.185088    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.185276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.185259    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnzd\" (UniqueName: \"kubernetes.io/projected/176cbbaa-756e-4daa-b257-7c38bd4463c7-kube-api-access-mlnzd\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.185351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.185325    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.360304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.306573    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnzd\" (UniqueName: \"kubernetes.io/projected/176cbbaa-756e-4daa-b257-7c38bd4463c7-kube-api-access-mlnzd\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.360304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.306667    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.360304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.306824    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.360304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.307809    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.360304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.322645    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnzd\" (UniqueName: \"kubernetes.io/projected/176cbbaa-756e-4daa-b257-7c38bd4463c7-kube-api-access-mlnzd\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.382589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.382554    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6de735-5rl2c\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.474968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.474939    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"]
Oct 31 01:44:11.487848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.487797    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:11.487964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.487913    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:11.488033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.487965    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:11.544297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.544254    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:11.906143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.896308    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" event={"ID":"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e","Type":"ContainerStarted","Data":"400339eb833ad4dcceac34087b942cd8b9ea5e74f1a4116f34a21c3066ae7906"}
Oct 31 01:44:11.906143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:11.896377    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" event={"ID":"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e","Type":"ContainerStarted","Data":"7c38805e8adb88a797bb2ef0901696df9b1d191b243795d56d86613ab569c3f0"}
Oct 31 01:44:11.916975 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:44:11.914810    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:44:12.036187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.034858    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" event={"ID":"6f54a9a0-1e40-4213-a73e-554d493f9313","Type":"ContainerStarted","Data":"3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434"}
Oct 31 01:44:12.036187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.034919    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" event={"ID":"6f54a9a0-1e40-4213-a73e-554d493f9313","Type":"ContainerStarted","Data":"906c0cdb160e0670e3ca8109ff7ebbaf86de8e54d3d7d77a0c74abdbfbdb45ad"}
Oct 31 01:44:12.089773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.089712    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podStartSLOduration=4.089691707 podStartE2EDuration="4.089691707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:12.018234654 +0000 UTC m=+1409.966883069" watchObservedRunningTime="2026-10-31 01:44:12.089691707 +0000 UTC m=+1410.038340104"
Oct 31 01:44:12.115313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.109950    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podStartSLOduration=3.109931121 podStartE2EDuration="3.109931121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:12.084290026 +0000 UTC m=+1410.032938430" watchObservedRunningTime="2026-10-31 01:44:12.109931121 +0000 UTC m=+1410.058579516"
Oct 31 01:44:12.151063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.119113    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:12.248133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.162901    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:12.248133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.184305    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:12.248133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.184170    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:12.248133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.184593    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:44:12.309099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.259164    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 01:44:12.309099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.259225    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 01:44:12.494628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.494392    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"]
Oct 31 01:44:12.594594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.594520    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:12.595966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.595638    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:12.595966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.595692    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:12.939913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:12.939883    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:13.014681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.014647    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5003abae-3098-4227-8b03-49f37e126502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5003abae-3098-4227-8b03-49f37e126502-maas-api-openshift-ca\") pod \"5003abae-3098-4227-8b03-49f37e126502\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") "
Oct 31 01:44:13.044238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.044207    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5003abae-3098-4227-8b03-49f37e126502-kube-api-access-kvk9v\" (UniqueName: \"kubernetes.io/projected/5003abae-3098-4227-8b03-49f37e126502-kube-api-access-kvk9v\") pod \"5003abae-3098-4227-8b03-49f37e126502\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") "
Oct 31 01:44:13.072296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.072209    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5003abae-3098-4227-8b03-49f37e126502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5003abae-3098-4227-8b03-49f37e126502-maas-api-xks-ca\") pod \"5003abae-3098-4227-8b03-49f37e126502\" (UID: \"5003abae-3098-4227-8b03-49f37e126502\") "
Oct 31 01:44:13.143525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.032009    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5003abae-3098-4227-8b03-49f37e126502-maas-api-openshift-ca" pod "5003abae-3098-4227-8b03-49f37e126502" (UID: "5003abae-3098-4227-8b03-49f37e126502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:13.144019 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.143987    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:13.144218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.144191    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:13.177839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.177795    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5003abae-3098-4227-8b03-49f37e126502-kube-api-access-kvk9v" pod "5003abae-3098-4227-8b03-49f37e126502" (UID: "5003abae-3098-4227-8b03-49f37e126502"). InnerVolumeSpecName "kube-api-access-kvk9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:13.198496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.198463    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 01:44:13.199148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.199120    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 01:44:13.199686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.198693    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:13.199819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.199800    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:44:13.208106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.208053    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c" event={"ID":"176cbbaa-756e-4daa-b257-7c38bd4463c7","Type":"ContainerStarted","Data":"df7b77671626a69503a71871548af3f7aded02f808f0b87020b9dc5261ea7f80"}
Oct 31 01:44:13.210649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.210619    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"
Oct 31 01:44:13.210898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.210878    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89" event={"ID":"5003abae-3098-4227-8b03-49f37e126502","Type":"ContainerDied","Data":"d9be322068ae4d51b9c673902d6924a3644ddedcaebfa3d0c5e39e6b5a8f5e1f"}
Oct 31 01:44:13.211062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.211026    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9be322068ae4d51b9c673902d6924a3644ddedcaebfa3d0c5e39e6b5a8f5e1f"
Oct 31 01:44:13.212236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.212212    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5003abae-3098-4227-8b03-49f37e126502-maas-api-xks-ca" pod "5003abae-3098-4227-8b03-49f37e126502" (UID: "5003abae-3098-4227-8b03-49f37e126502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:13.226210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.226184    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5003abae-3098-4227-8b03-49f37e126502-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:13.230068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.230030    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvk9v\" (UniqueName: \"kubernetes.io/projected/5003abae-3098-4227-8b03-49f37e126502-kube-api-access-kvk9v\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:13.230246 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.230221    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5003abae-3098-4227-8b03-49f37e126502-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:13.594600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.594563    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:13.595124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:13.594627    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:14.186066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.185445    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 01:44:14.186066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.185508    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 01:44:14.188498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.188468    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:14.188710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.188681    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:14.222355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.222321    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:14.222618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.222590    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:44:14.289415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.289377    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c" event={"ID":"176cbbaa-756e-4daa-b257-7c38bd4463c7","Type":"ContainerStarted","Data":"0c7468e2537178f18dbf952bc3dcda86026d22f3096b401aa405009e27d38db7"}
Oct 31 01:44:14.599612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.599517    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:14.610067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:14.608185    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:15.127064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.126418    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:15.127064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.126477    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:15.308066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.301685    5293 generic.go:356] "Generic (PLEG): container finished" podID="176cbbaa-756e-4daa-b257-7c38bd4463c7" containerID="0c7468e2537178f18dbf952bc3dcda86026d22f3096b401aa405009e27d38db7" exitCode=0
Oct 31 01:44:15.308066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.301937    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c" event={"ID":"176cbbaa-756e-4daa-b257-7c38bd4463c7","Type":"ContainerDied","Data":"0c7468e2537178f18dbf952bc3dcda86026d22f3096b401aa405009e27d38db7"}
Oct 31 01:44:15.315515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.315445    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:15.316918 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.316738    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:15.326310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.326285    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 01:44:15.680067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.650432    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:15.680067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.650485    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:15.775014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.773698    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:15.775200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.775079    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:15.776567 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:15.776551    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 01:44:16.149094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.128464    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:16.149094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.128524    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:16.289825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.289777    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:16.391282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.390866    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c" event={"ID":"176cbbaa-756e-4daa-b257-7c38bd4463c7","Type":"ContainerDied","Data":"df7b77671626a69503a71871548af3f7aded02f808f0b87020b9dc5261ea7f80"}
Oct 31 01:44:16.391282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.390983    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df7b77671626a69503a71871548af3f7aded02f808f0b87020b9dc5261ea7f80"
Oct 31 01:44:16.392211 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.392179    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"
Oct 31 01:44:16.451068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.437521    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-openshift-ca\") pod \"176cbbaa-756e-4daa-b257-7c38bd4463c7\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") "
Oct 31 01:44:16.451469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.451442    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176cbbaa-756e-4daa-b257-7c38bd4463c7-kube-api-access-mlnzd\" (UniqueName: \"kubernetes.io/projected/176cbbaa-756e-4daa-b257-7c38bd4463c7-kube-api-access-mlnzd\") pod \"176cbbaa-756e-4daa-b257-7c38bd4463c7\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") "
Oct 31 01:44:16.451651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.451635    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-xks-ca\") pod \"176cbbaa-756e-4daa-b257-7c38bd4463c7\" (UID: \"176cbbaa-756e-4daa-b257-7c38bd4463c7\") "
Oct 31 01:44:16.454460 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.454427    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-openshift-ca" pod "176cbbaa-756e-4daa-b257-7c38bd4463c7" (UID: "176cbbaa-756e-4daa-b257-7c38bd4463c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:16.464575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.464545    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176cbbaa-756e-4daa-b257-7c38bd4463c7-kube-api-access-mlnzd" pod "176cbbaa-756e-4daa-b257-7c38bd4463c7" (UID: "176cbbaa-756e-4daa-b257-7c38bd4463c7"). InnerVolumeSpecName "kube-api-access-mlnzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:16.466904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.466879    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-xks-ca" pod "176cbbaa-756e-4daa-b257-7c38bd4463c7" (UID: "176cbbaa-756e-4daa-b257-7c38bd4463c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:16.558881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.558539    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlnzd\" (UniqueName: \"kubernetes.io/projected/176cbbaa-756e-4daa-b257-7c38bd4463c7-kube-api-access-mlnzd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:16.558881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.558573    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:16.558881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.558590    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176cbbaa-756e-4daa-b257-7c38bd4463c7-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:16.594351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.594321    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:16.594584 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:16.594557    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:17.116123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.115517    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:17.116123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.115576    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:17.669482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.669173    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"]
Oct 31 01:44:17.670580 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.670551    5293 cpu_manager.go:414] "containerMap: removing container" podUID="176cbbaa-756e-4daa-b257-7c38bd4463c7" containerName="revoke-keys"
Oct 31 01:44:17.670735 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.670722    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176cbbaa-756e-4daa-b257-7c38bd4463c7" containerName="revoke-keys"
Oct 31 01:44:17.670857 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.670847    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5003abae-3098-4227-8b03-49f37e126502" containerName="revoke-keys"
Oct 31 01:44:17.670932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.670922    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5003abae-3098-4227-8b03-49f37e126502" containerName="revoke-keys"
Oct 31 01:44:17.671216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.671202    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="176cbbaa-756e-4daa-b257-7c38bd4463c7" containerName="revoke-keys"
Oct 31 01:44:17.671341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.671331    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5003abae-3098-4227-8b03-49f37e126502" containerName="revoke-keys"
Oct 31 01:44:17.671520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.671342    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:17.671588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.671558    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:17.675914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.675503    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"]
Oct 31 01:44:17.675914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.675585    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.680021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.680004    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:44:17.680245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.680232    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:44:17.680366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.680343    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:17.680425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.680416    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:17.769946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.769910    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xst2\" (UniqueName: \"kubernetes.io/projected/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-kube-api-access-4xst2\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.770163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.770064    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.770235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.770216    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.859361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.859284    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:44:17.871763 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.871730    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xst2\" (UniqueName: \"kubernetes.io/projected/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-kube-api-access-4xst2\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.872006 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.871991    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.872208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.872195    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.872738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.872717    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.874736 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.874719    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.883248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.883231    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xst2\" (UniqueName: \"kubernetes.io/projected/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-kube-api-access-4xst2\") pod \"maas-api-revoke-sub-baaff0ba-9mqb2\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:17.991902 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:17.991809    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:18.121062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.119441    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:18.121062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.119497    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:18.463243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.463208    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"]
Oct 31 01:44:18.483012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.482972    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"]
Oct 31 01:44:18.483527 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.483128    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.486537 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.486513    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tls-certs\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.486807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.486784    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjxz\" (UniqueName: \"kubernetes.io/projected/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-kube-api-access-xrjxz\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.486980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.486963    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tmp-dir\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.487474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.487441    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-model-cache\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.487645 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.487620    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-dshm\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.487920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.487883    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-home\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.493915 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.493896    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"distinct-w3-7e5955-kserve-self-signed-certs\""
Oct 31 01:44:18.591692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.591632    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjxz\" (UniqueName: \"kubernetes.io/projected/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-kube-api-access-xrjxz\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.592014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.591986    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tmp-dir\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.592296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.592275    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-model-cache\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.592430 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.592414    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-dshm\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.592621 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.592606    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-home\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.592769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.592753    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tls-certs\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.716837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.675156    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:18.716837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.675232    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:18.716837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.676319    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-model-cache\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.716837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.677820    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tmp-dir\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.716837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.686445    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"]
Oct 31 01:44:18.716837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.686857    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-home\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.777299 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:44:18.726617    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bdab48_e65e_4dc0_94e9_e6d0458d8a90.slice/crio-c8c1de428d59c3b72eefd025e41550e6af26a8edd4bd73ad48a5b2112192b4b2 WatchSource:0}: Error finding container c8c1de428d59c3b72eefd025e41550e6af26a8edd4bd73ad48a5b2112192b4b2: Status 404 returned error can't find the container with id c8c1de428d59c3b72eefd025e41550e6af26a8edd4bd73ad48a5b2112192b4b2
Oct 31 01:44:18.780835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.780807    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tls-certs\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.788415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.788383    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-dshm\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.830631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.829791    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjxz\" (UniqueName: \"kubernetes.io/projected/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-kube-api-access-xrjxz\") pod \"distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:18.848938 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:18.848630    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:19.118087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:19.117584    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:19.118087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:19.117645    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:19.440408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:19.430910    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"]
Oct 31 01:44:19.594257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:19.578498    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2" event={"ID":"22bdab48-e65e-4dc0-94e9-e6d0458d8a90","Type":"ContainerStarted","Data":"c8c1de428d59c3b72eefd025e41550e6af26a8edd4bd73ad48a5b2112192b4b2"}
Oct 31 01:44:19.598062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:19.594482    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:19.598062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:19.594557    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:19.962074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:19.956102    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"]
Oct 31 01:44:19.988242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:19.970507    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.083743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.078030    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.083743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.082684    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.083743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.082856    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zk82\" (UniqueName: \"kubernetes.io/projected/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-kube-api-access-5zk82\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.142170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.127274    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:20.142170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.127330    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:20.208114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.200482    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.208114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.200641    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zk82\" (UniqueName: \"kubernetes.io/projected/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-kube-api-access-5zk82\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.208114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.200807    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.208114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.202022    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.263060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.262405    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"]
Oct 31 01:44:20.456891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.455200    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zk82\" (UniqueName: \"kubernetes.io/projected/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-kube-api-access-5zk82\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.456891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.455362    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:20.555148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.530377    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:20.618083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.601469    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91311875-5w4nc\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.618083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.602799    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:20.618083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.602864    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:20.805067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.778119    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:20.805067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.779221    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:20.805067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.780563    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:20.893409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.881660    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:20.916723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.916681    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" event={"ID":"ccb5b30f-e8ae-48ed-beb8-caec78c713f3","Type":"ContainerStarted","Data":"f1cfc0a372f997ea71f181464bbcf9e02905cc2dae9c7c223a64ea3ac4c81ae0"}
Oct 31 01:44:20.968618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.968556    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:20.972392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:20.972363    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2" event={"ID":"22bdab48-e65e-4dc0-94e9-e6d0458d8a90","Type":"ContainerStarted","Data":"59804c338a12beb1f71df8e6ae49ebdcb5438e00bcde852bbcf838f6fb60364d"}
Oct 31 01:44:21.016073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.005087    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.245066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.213029    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:21.245066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.244408    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:21.605816 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.572136    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.605816 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.572577    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:44:21.874123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.861899    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:44:21.892688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.874422    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:21.892688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.874843    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:44:21.892688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.874903    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:44:21.914089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.896157    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.914089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.896585    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:44:21.914089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.909128    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:21.914089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.909180    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:21.989068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:21.972491    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.007629 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.007597    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:44:22.032207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.017172    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.100106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.100067    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:44:22.225890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.126610    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:44:22.254458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.175091    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:22.254672 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.254629    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 01:44:22.268230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.201139    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.350867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.350819    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:22.351176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.351132    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.379514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.379479    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.835174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.782455    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:44:22.927108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.916987    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.952102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.930928    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:22.952102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.930987    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.952102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.941194    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:22.952102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.941245    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:44:22.960130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.960085    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.984061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.980286    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.995403 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:22.995360    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.042222 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.039988    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:44:23.056069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.052854    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.273079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.260989    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.273079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.261951    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.273079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.264404    5293 generic.go:356] "Generic (PLEG): container finished" podID="22bdab48-e65e-4dc0-94e9-e6d0458d8a90" containerID="59804c338a12beb1f71df8e6ae49ebdcb5438e00bcde852bbcf838f6fb60364d" exitCode=0
Oct 31 01:44:23.360065 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:44:23.359632    5293 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.133s"
Oct 31 01:44:23.386069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.367251    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:44:23.386069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.367314    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2" event={"ID":"22bdab48-e65e-4dc0-94e9-e6d0458d8a90","Type":"ContainerDied","Data":"59804c338a12beb1f71df8e6ae49ebdcb5438e00bcde852bbcf838f6fb60364d"}
Oct 31 01:44:23.386069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.367818    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:44:23.386069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.367871    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:44:23.386069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.367918    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"]
Oct 31 01:44:23.742130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.738302    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:44:23.783056 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.782999    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.929299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.917671    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.929299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:23.919155    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:44:24.081288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.077018    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:44:24.117203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.117174    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:24.365725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.365656    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:24.387063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.366084    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:24.387063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.375390    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 31 01:44:24.387063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.375464    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:24.387063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.375724    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:44:24.410210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.403172    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:24.680103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.674912    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:24.680103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.674987    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:24.753514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.752270    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:24.760463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.760415    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:24.760606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.760484    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:24.933421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.920821    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:24.951355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.951323    5293 generic.go:356] "Generic (PLEG): container finished" podID="8443aee4-f3ae-42f5-8e1d-2b7a10e052e7" containerID="d5932834b2b4f2dc96db442fb2f2494cbef694c0ad9a1fc6b567711ec4fbae5b" exitCode=0
Oct 31 01:44:24.966424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.965939    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:24.966424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.966065    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:44:24.966424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.966148    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" event={"ID":"ccb5b30f-e8ae-48ed-beb8-caec78c713f3","Type":"ContainerStarted","Data":"09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226"}
Oct 31 01:44:24.966424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.966173    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc" event={"ID":"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7","Type":"ContainerDied","Data":"d5932834b2b4f2dc96db442fb2f2494cbef694c0ad9a1fc6b567711ec4fbae5b"}
Oct 31 01:44:24.966424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.966211    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc" event={"ID":"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7","Type":"ContainerStarted","Data":"3b6e51d043e99a692715213091c54822a891d8d8aae2966ffd07261964606fee"}
Oct 31 01:44:24.967799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.966267    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:44:24.969578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.969558    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:44:24.969765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.969752    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:44:24.969916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.969905    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:44:24.970062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.970032    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:44:24.970197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.970186    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:24.980010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.979960    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podStartSLOduration=6.979943224 podStartE2EDuration="6.979943224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:24.967979292 +0000 UTC m=+1422.916627687" watchObservedRunningTime="2026-10-31 01:44:24.979943224 +0000 UTC m=+1422.928591617"
Oct 31 01:44:24.983135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.981987    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:44:24.983135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.982056    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:44:24.983329 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.983255    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:44:24.983943 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.983547    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:44:24.985388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:24.985370    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:25.062084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.062027    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-xks-ca\") pod \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") "
Oct 31 01:44:25.062260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.062124    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-kube-api-access-4xst2\" (UniqueName: \"kubernetes.io/projected/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-kube-api-access-4xst2\") pod \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") "
Oct 31 01:44:25.062366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.062350    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-openshift-ca\") pod \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\" (UID: \"22bdab48-e65e-4dc0-94e9-e6d0458d8a90\") "
Oct 31 01:44:25.065145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.065110    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-xks-ca" pod "22bdab48-e65e-4dc0-94e9-e6d0458d8a90" (UID: "22bdab48-e65e-4dc0-94e9-e6d0458d8a90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:25.065263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.065199    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-openshift-ca" pod "22bdab48-e65e-4dc0-94e9-e6d0458d8a90" (UID: "22bdab48-e65e-4dc0-94e9-e6d0458d8a90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:25.065402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.065365    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-kube-api-access-4xst2" pod "22bdab48-e65e-4dc0-94e9-e6d0458d8a90" (UID: "22bdab48-e65e-4dc0-94e9-e6d0458d8a90"). InnerVolumeSpecName "kube-api-access-4xst2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:25.173604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.166272    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:25.173604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.166307    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xst2\" (UniqueName: \"kubernetes.io/projected/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-kube-api-access-4xst2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:25.173604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.166323    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bdab48-e65e-4dc0-94e9-e6d0458d8a90-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:25.716251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.716201    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:25.716464 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.716268    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:25.852245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.852176    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:25.971528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.971441    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2" event={"ID":"22bdab48-e65e-4dc0-94e9-e6d0458d8a90","Type":"ContainerDied","Data":"c8c1de428d59c3b72eefd025e41550e6af26a8edd4bd73ad48a5b2112192b4b2"}
Oct 31 01:44:25.971528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.971500    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8c1de428d59c3b72eefd025e41550e6af26a8edd4bd73ad48a5b2112192b4b2"
Oct 31 01:44:25.971528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:25.971451    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"
Oct 31 01:44:26.400117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.398113    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:26.476089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.475309    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:26.476966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.476943    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:44:26.509448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.509410    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-kube-api-access-5zk82\" (UniqueName: \"kubernetes.io/projected/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-kube-api-access-5zk82\") pod \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") "
Oct 31 01:44:26.509643 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.509549    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-xks-ca\") pod \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") "
Oct 31 01:44:26.509730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.509714    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-openshift-ca\") pod \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\" (UID: \"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7\") "
Oct 31 01:44:26.510140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.510108    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-openshift-ca" pod "8443aee4-f3ae-42f5-8e1d-2b7a10e052e7" (UID: "8443aee4-f3ae-42f5-8e1d-2b7a10e052e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:26.510497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.510475    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:26.511808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.511785    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-xks-ca" pod "8443aee4-f3ae-42f5-8e1d-2b7a10e052e7" (UID: "8443aee4-f3ae-42f5-8e1d-2b7a10e052e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:26.511897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.511878    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-kube-api-access-5zk82" pod "8443aee4-f3ae-42f5-8e1d-2b7a10e052e7" (UID: "8443aee4-f3ae-42f5-8e1d-2b7a10e052e7"). InnerVolumeSpecName "kube-api-access-5zk82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:26.612284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.612193    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zk82\" (UniqueName: \"kubernetes.io/projected/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-kube-api-access-5zk82\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:26.612284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.612229    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:26.980371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.980344    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"
Oct 31 01:44:26.980823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.980389    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc" event={"ID":"8443aee4-f3ae-42f5-8e1d-2b7a10e052e7","Type":"ContainerDied","Data":"3b6e51d043e99a692715213091c54822a891d8d8aae2966ffd07261964606fee"}
Oct 31 01:44:26.980823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:26.980455    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b6e51d043e99a692715213091c54822a891d8d8aae2966ffd07261964606fee"
Oct 31 01:44:28.854252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:28.852171    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:28.905386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:28.905345    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:29.043937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:29.043908    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:30.750505 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.750469    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"]
Oct 31 01:44:30.751153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.751124    5293 cpu_manager.go:414] "containerMap: removing container" podUID="22bdab48-e65e-4dc0-94e9-e6d0458d8a90" containerName="revoke-keys"
Oct 31 01:44:30.751153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.751144    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22bdab48-e65e-4dc0-94e9-e6d0458d8a90" containerName="revoke-keys"
Oct 31 01:44:30.751326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.751194    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8443aee4-f3ae-42f5-8e1d-2b7a10e052e7" containerName="revoke-keys"
Oct 31 01:44:30.751326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.751202    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8443aee4-f3ae-42f5-8e1d-2b7a10e052e7" containerName="revoke-keys"
Oct 31 01:44:30.751421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.751359    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="22bdab48-e65e-4dc0-94e9-e6d0458d8a90" containerName="revoke-keys"
Oct 31 01:44:30.751421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.751378    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8443aee4-f3ae-42f5-8e1d-2b7a10e052e7" containerName="revoke-keys"
Oct 31 01:44:30.754748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.754728    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:30.760052 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.760016    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:44:30.760256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.760227    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:30.760349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.760314    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:44:30.760416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.760376    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:30.765530 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.765508    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"]
Oct 31 01:44:30.880555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.880517    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfxh\" (UniqueName: \"kubernetes.io/projected/9b96cbc5-8856-4892-9856-1775e4985c6b-kube-api-access-fwfxh\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:30.880914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.880889    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:30.881030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.881015    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:30.984888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.984856    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfxh\" (UniqueName: \"kubernetes.io/projected/9b96cbc5-8856-4892-9856-1775e4985c6b-kube-api-access-fwfxh\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:30.985053 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.984981    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:30.985053 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.985011    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:30.985682 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.985641    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:30.987538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:30.987519    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:31.001369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:31.001318    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfxh\" (UniqueName: \"kubernetes.io/projected/9b96cbc5-8856-4892-9856-1775e4985c6b-kube-api-access-fwfxh\") pod \"maas-api-revoke-sub-bf054a0b-6qn92\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:31.070205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:31.070175    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:31.356032 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:31.356004    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"]
Oct 31 01:44:31.357009 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:44:31.356978    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b96cbc5_8856_4892_9856_1775e4985c6b.slice/crio-3290c9839269ad58d783ad2ffceeab0f399715c235329088e4548e3a675f1ca0 WatchSource:0}: Error finding container 3290c9839269ad58d783ad2ffceeab0f399715c235329088e4548e3a675f1ca0: Status 404 returned error can't find the container with id 3290c9839269ad58d783ad2ffceeab0f399715c235329088e4548e3a675f1ca0
Oct 31 01:44:32.059884 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:32.059849    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92" event={"ID":"9b96cbc5-8856-4892-9856-1775e4985c6b","Type":"ContainerStarted","Data":"e37f746c588f5fb59d7fb8507b1e8f537c48f3b046c327671b1620ec20059976"}
Oct 31 01:44:32.060269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:32.059905    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92" event={"ID":"9b96cbc5-8856-4892-9856-1775e4985c6b","Type":"ContainerStarted","Data":"3290c9839269ad58d783ad2ffceeab0f399715c235329088e4548e3a675f1ca0"}
Oct 31 01:44:32.088342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:32.088073    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92" podStartSLOduration=2.088032337 podStartE2EDuration="2.088032337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:32.079650886 +0000 UTC m=+1430.028299277" watchObservedRunningTime="2026-10-31 01:44:32.088032337 +0000 UTC m=+1430.036680730"
Oct 31 01:44:33.193653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:33.193614    5293 generic.go:356] "Generic (PLEG): container finished" podID="9b96cbc5-8856-4892-9856-1775e4985c6b" containerID="e37f746c588f5fb59d7fb8507b1e8f537c48f3b046c327671b1620ec20059976" exitCode=0
Oct 31 01:44:33.194486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:33.194451    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92" event={"ID":"9b96cbc5-8856-4892-9856-1775e4985c6b","Type":"ContainerDied","Data":"e37f746c588f5fb59d7fb8507b1e8f537c48f3b046c327671b1620ec20059976"}
Oct 31 01:44:33.895067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:33.894139    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:33.940066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:33.937197    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.009399 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.008146    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.009646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.009615    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.044008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.042888    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.062830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.062790    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.098330 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.098292    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.385067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.380116    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.659181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.659126    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded"
Oct 31 01:44:34.739161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.739115    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.796197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.796148    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.902876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.900626    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.902876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.901468    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:44:34.951246 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.951151    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.952288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:34.952264    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:44:35.029064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.015150    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:35.029064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.017372    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:44:35.029064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.018930    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.046064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.029537    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:44:35.061104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.046345    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.061104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.046869    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:35.279333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.279233    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:35.310066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.281982    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:44:35.476061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.310348    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.476061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.311101    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:44:35.476061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.470660    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:44:35.830066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.813578    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.338068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.905205    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.338068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.905918    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.338068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:35.905964    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:44:36.338068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.021766    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.342507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.025449    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:44:36.342507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.038473    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.382089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.064626    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:44:36.382089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.064726    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:36.382089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.343066    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.382089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.292197    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:36.382089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.090718    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:36.382089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.088588    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:36.382089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.343335    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.499107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.495575    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.572937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.572897    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b96cbc5-8856-4892-9856-1775e4985c6b-kube-api-access-fwfxh\" (UniqueName: \"kubernetes.io/projected/9b96cbc5-8856-4892-9856-1775e4985c6b-kube-api-access-fwfxh\") pod \"9b96cbc5-8856-4892-9856-1775e4985c6b\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") "
Oct 31 01:44:36.650318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.650289    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-xks-ca\") pod \"9b96cbc5-8856-4892-9856-1775e4985c6b\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") "
Oct 31 01:44:36.666076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.654222    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-openshift-ca\") pod \"9b96cbc5-8856-4892-9856-1775e4985c6b\" (UID: \"9b96cbc5-8856-4892-9856-1775e4985c6b\") "
Oct 31 01:44:36.675170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.611088    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:44:36.675944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.675900    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-openshift-ca" pod "9b96cbc5-8856-4892-9856-1775e4985c6b" (UID: "9b96cbc5-8856-4892-9856-1775e4985c6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:36.732068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.728170    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-s9rxb" podUID="e75104bf-ae55-488f-bab5-595defac1e17" containerName="server" probeResult="failure" output="Get \"https://10.133.0.20:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.769332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.769300    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:44:36.769635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.769618    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:44:36.852466 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.851404    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b96cbc5-8856-4892-9856-1775e4985c6b-kube-api-access-fwfxh" pod "9b96cbc5-8856-4892-9856-1775e4985c6b" (UID: "9b96cbc5-8856-4892-9856-1775e4985c6b"). InnerVolumeSpecName "kube-api-access-fwfxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:36.854115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.853851    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfxh\" (UniqueName: \"kubernetes.io/projected/9b96cbc5-8856-4892-9856-1775e4985c6b-kube-api-access-fwfxh\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:36.854115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.853900    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:36.887315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.887275    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-xks-ca" pod "9b96cbc5-8856-4892-9856-1775e4985c6b" (UID: "9b96cbc5-8856-4892-9856-1775e4985c6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:36.956061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:36.955952    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b96cbc5-8856-4892-9856-1775e4985c6b-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:37.185068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.178506    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92" event={"ID":"9b96cbc5-8856-4892-9856-1775e4985c6b","Type":"ContainerDied","Data":"3290c9839269ad58d783ad2ffceeab0f399715c235329088e4548e3a675f1ca0"}
Oct 31 01:44:37.185068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.178591    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3290c9839269ad58d783ad2ffceeab0f399715c235329088e4548e3a675f1ca0"
Oct 31 01:44:37.185068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.178682    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"
Oct 31 01:44:37.299122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.298526    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.320531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.320474    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:44:37.321155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.321119    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.326329 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.321346    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.326790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.321396    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.349810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.349771    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:44:37.358030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.357998    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:37.369078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.359350    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:44:37.370007 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.369974    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:37.403134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.377865    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:37.403134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.377925    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.499211 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:37.499163    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.038209 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.038160    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.312151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.303144    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.326656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.326613    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:44:38.328098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.328063    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.354397 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.354359    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.386623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.386581    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.387157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.387124    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:44:38.396065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.396006    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.404067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.396861    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:38.444272 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.444230    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.446018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.445979    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:38.446287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.446245    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.512071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.505145    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.781189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:38.781135    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 31 01:44:39.307371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.307335    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:39.315447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.315420    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:44:39.336075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.330132    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:44:39.336075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.334141    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:39.385064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.354997    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:39.425803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.416636    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:39.426652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.426617    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:39.435563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.435533    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:39.454782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.454758    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:44:39.535004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.534974    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:44:39.535647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.535627    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:44:39.567302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.565771    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:44:39.579416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.579393    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:44:39.580687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.580670    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:44:39.580866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.580856    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:39.591084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.586160    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:44:39.591084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.586384    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:44:39.748354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.748320    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"]
Oct 31 01:44:39.748923 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.748906    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9b96cbc5-8856-4892-9856-1775e4985c6b" containerName="revoke-keys"
Oct 31 01:44:39.748923 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.748924    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b96cbc5-8856-4892-9856-1775e4985c6b" containerName="revoke-keys"
Oct 31 01:44:39.749087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.749031    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b96cbc5-8856-4892-9856-1775e4985c6b" containerName="revoke-keys"
Oct 31 01:44:39.753880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.753855    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.763374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.763347    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"distinct-2-w3-7e5955-kserve-self-signed-certs\""
Oct 31 01:44:39.764918 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.764892    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"]
Oct 31 01:44:39.787713 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.787653    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-dshm\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.787889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.787758    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-model-cache\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.787889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.787830    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tls-certs\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.787991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.787928    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvwg\" (UniqueName: \"kubernetes.io/projected/5990f411-ac1a-405b-ab45-5ea2903a3fa0-kube-api-access-mlvwg\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.788115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.788059    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tmp-dir\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.788306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.788285    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-home\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.894119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.890966    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-home\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.894119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.891033    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-dshm\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.894119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.891077    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-model-cache\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.894119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.891112    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tls-certs\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.894119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.891224    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvwg\" (UniqueName: \"kubernetes.io/projected/5990f411-ac1a-405b-ab45-5ea2903a3fa0-kube-api-access-mlvwg\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.894119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.891268    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tmp-dir\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.894119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.891931    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tmp-dir\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.894119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.892220    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-home\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.912409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.912374    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-model-cache\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.915462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.915433    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-dshm\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.967066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.938662    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tls-certs\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:39.968000 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:39.962940    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvwg\" (UniqueName: \"kubernetes.io/projected/5990f411-ac1a-405b-ab45-5ea2903a3fa0-kube-api-access-mlvwg\") pod \"distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") " pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:40.075579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.075336    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:40.396559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.396519    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"]
Oct 31 01:44:40.469292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.469272    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"]
Oct 31 01:44:40.469463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.469451    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"]
Oct 31 01:44:40.469677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.469664    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.474908 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.474890    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:40.475150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.475127    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:40.475635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.475615    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:44:40.476253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.476134    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:44:40.602145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.602103    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.602601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.602148    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fwr\" (UniqueName: \"kubernetes.io/projected/99dd44dc-6599-4ec1-812d-d8e4b7144e09-kube-api-access-84fwr\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.602601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.602278    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.703824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.703733    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.703824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.703767    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84fwr\" (UniqueName: \"kubernetes.io/projected/99dd44dc-6599-4ec1-812d-d8e4b7144e09-kube-api-access-84fwr\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.703824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.703807    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.704501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.704478    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.706245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.706222    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.715586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.715565    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fwr\" (UniqueName: \"kubernetes.io/projected/99dd44dc-6599-4ec1-812d-d8e4b7144e09-kube-api-access-84fwr\") pod \"maas-api-revoke-sub-d9125bc4-c9l7d\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.794741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.794713    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:40.869178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.869150    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"]
Oct 31 01:44:40.873635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.873614    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:40.898920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:40.898883    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"]
Oct 31 01:44:41.008758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.008668    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.008894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.008863    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swftr\" (UniqueName: \"kubernetes.io/projected/d02964b6-a6cc-495d-98ee-9b8bff09cadc-kube-api-access-swftr\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.008944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.008903    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.080980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.080949    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"]
Oct 31 01:44:41.081828 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:44:41.081798    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99dd44dc_6599_4ec1_812d_d8e4b7144e09.slice/crio-497166d8a65e58499596e7af8e1aa01977ba1487fcd487f4d9468fa7e2dbf6ab WatchSource:0}: Error finding container 497166d8a65e58499596e7af8e1aa01977ba1487fcd487f4d9468fa7e2dbf6ab: Status 404 returned error can't find the container with id 497166d8a65e58499596e7af8e1aa01977ba1487fcd487f4d9468fa7e2dbf6ab
Oct 31 01:44:41.109676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.109651    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.109803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.109753    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swftr\" (UniqueName: \"kubernetes.io/projected/d02964b6-a6cc-495d-98ee-9b8bff09cadc-kube-api-access-swftr\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.109803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.109778    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.110314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.110286    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.112333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.112311    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.125493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.125473    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swftr\" (UniqueName: \"kubernetes.io/projected/d02964b6-a6cc-495d-98ee-9b8bff09cadc-kube-api-access-swftr\") pod \"maas-api-revoke-sub-599d516a-r8gpn\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.201299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.201271    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:41.411414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.411102    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" event={"ID":"5990f411-ac1a-405b-ab45-5ea2903a3fa0","Type":"ContainerStarted","Data":"cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2"}
Oct 31 01:44:41.411414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.411153    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" event={"ID":"5990f411-ac1a-405b-ab45-5ea2903a3fa0","Type":"ContainerStarted","Data":"de0c34046de2a6474de997be30b8065e8ad90387296a7fa18f56625b19dc80dd"}
Oct 31 01:44:41.412699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.412672    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d" event={"ID":"99dd44dc-6599-4ec1-812d-d8e4b7144e09","Type":"ContainerStarted","Data":"497166d8a65e58499596e7af8e1aa01977ba1487fcd487f4d9468fa7e2dbf6ab"}
Oct 31 01:44:41.433924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.433872    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podStartSLOduration=2.433860153 podStartE2EDuration="2.433860153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:41.429392609 +0000 UTC m=+1439.378041001" watchObservedRunningTime="2026-10-31 01:44:41.433860153 +0000 UTC m=+1439.382508547"
Oct 31 01:44:41.486875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:41.486850    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"]
Oct 31 01:44:41.487934 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:44:41.487908    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02964b6_a6cc_495d_98ee_9b8bff09cadc.slice/crio-550fda24b977809c7c151aa8baa685d48517274b4ed6132ec3f7f1278108b856 WatchSource:0}: Error finding container 550fda24b977809c7c151aa8baa685d48517274b4ed6132ec3f7f1278108b856: Status 404 returned error can't find the container with id 550fda24b977809c7c151aa8baa685d48517274b4ed6132ec3f7f1278108b856
Oct 31 01:44:41.778243 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:44:41.778149    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:44:42.084643 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:42.083135    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:42.434458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:42.434414    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn" event={"ID":"d02964b6-a6cc-495d-98ee-9b8bff09cadc","Type":"ContainerStarted","Data":"044208b6e46f48342da2d020b2c59375bb7c882d769cb2a1ab430eb207508318"}
Oct 31 01:44:42.434656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:42.434521    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn" event={"ID":"d02964b6-a6cc-495d-98ee-9b8bff09cadc","Type":"ContainerStarted","Data":"550fda24b977809c7c151aa8baa685d48517274b4ed6132ec3f7f1278108b856"}
Oct 31 01:44:42.436181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:42.436151    5293 generic.go:356] "Generic (PLEG): container finished" podID="99dd44dc-6599-4ec1-812d-d8e4b7144e09" containerID="dccb29472cd39bb4bf4586dea2b105c911b72d4870ba13cbe41df9fce018a6d2" exitCode=0
Oct 31 01:44:42.436287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:42.436224    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d" event={"ID":"99dd44dc-6599-4ec1-812d-d8e4b7144e09","Type":"ContainerDied","Data":"dccb29472cd39bb4bf4586dea2b105c911b72d4870ba13cbe41df9fce018a6d2"}
Oct 31 01:44:43.471133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.469759    5293 generic.go:356] "Generic (PLEG): container finished" podID="d02964b6-a6cc-495d-98ee-9b8bff09cadc" containerID="044208b6e46f48342da2d020b2c59375bb7c882d769cb2a1ab430eb207508318" exitCode=0
Oct 31 01:44:43.471133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.469917    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn" event={"ID":"d02964b6-a6cc-495d-98ee-9b8bff09cadc","Type":"ContainerDied","Data":"044208b6e46f48342da2d020b2c59375bb7c882d769cb2a1ab430eb207508318"}
Oct 31 01:44:43.651656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.651624    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:43.702348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.702318    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:43.737001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.736928    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02964b6-a6cc-495d-98ee-9b8bff09cadc-kube-api-access-swftr\" (UniqueName: \"kubernetes.io/projected/d02964b6-a6cc-495d-98ee-9b8bff09cadc-kube-api-access-swftr\") pod \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") "
Oct 31 01:44:43.737001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.736991    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-xks-ca\") pod \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") "
Oct 31 01:44:43.737245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.737152    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-openshift-ca\") pod \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") "
Oct 31 01:44:43.737245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.737201    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99dd44dc-6599-4ec1-812d-d8e4b7144e09-kube-api-access-84fwr\" (UniqueName: \"kubernetes.io/projected/99dd44dc-6599-4ec1-812d-d8e4b7144e09-kube-api-access-84fwr\") pod \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") "
Oct 31 01:44:43.737364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.737263    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-openshift-ca\") pod \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\" (UID: \"d02964b6-a6cc-495d-98ee-9b8bff09cadc\") "
Oct 31 01:44:43.737364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.737327    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-xks-ca\") pod \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\" (UID: \"99dd44dc-6599-4ec1-812d-d8e4b7144e09\") "
Oct 31 01:44:43.738275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.737764    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-openshift-ca" pod "d02964b6-a6cc-495d-98ee-9b8bff09cadc" (UID: "d02964b6-a6cc-495d-98ee-9b8bff09cadc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:43.738275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.738061    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-openshift-ca" pod "99dd44dc-6599-4ec1-812d-d8e4b7144e09" (UID: "99dd44dc-6599-4ec1-812d-d8e4b7144e09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:43.738275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.738095    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.740187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.740151    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-xks-ca" pod "d02964b6-a6cc-495d-98ee-9b8bff09cadc" (UID: "d02964b6-a6cc-495d-98ee-9b8bff09cadc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:43.740699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.740671    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02964b6-a6cc-495d-98ee-9b8bff09cadc-kube-api-access-swftr" pod "d02964b6-a6cc-495d-98ee-9b8bff09cadc" (UID: "d02964b6-a6cc-495d-98ee-9b8bff09cadc"). InnerVolumeSpecName "kube-api-access-swftr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:43.740801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.740735    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99dd44dc-6599-4ec1-812d-d8e4b7144e09-kube-api-access-84fwr" pod "99dd44dc-6599-4ec1-812d-d8e4b7144e09" (UID: "99dd44dc-6599-4ec1-812d-d8e4b7144e09"). InnerVolumeSpecName "kube-api-access-84fwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:43.741160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.741139    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-xks-ca" pod "99dd44dc-6599-4ec1-812d-d8e4b7144e09" (UID: "99dd44dc-6599-4ec1-812d-d8e4b7144e09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:43.839592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.839556    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.839592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.839590    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84fwr\" (UniqueName: \"kubernetes.io/projected/99dd44dc-6599-4ec1-812d-d8e4b7144e09-kube-api-access-84fwr\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.839794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.839601    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99dd44dc-6599-4ec1-812d-d8e4b7144e09-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.839794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.839610    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swftr\" (UniqueName: \"kubernetes.io/projected/d02964b6-a6cc-495d-98ee-9b8bff09cadc-kube-api-access-swftr\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.839794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:43.839619    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02964b6-a6cc-495d-98ee-9b8bff09cadc-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.479676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.479593    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"
Oct 31 01:44:44.479676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.479630    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn" event={"ID":"d02964b6-a6cc-495d-98ee-9b8bff09cadc","Type":"ContainerDied","Data":"550fda24b977809c7c151aa8baa685d48517274b4ed6132ec3f7f1278108b856"}
Oct 31 01:44:44.480199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.479684    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="550fda24b977809c7c151aa8baa685d48517274b4ed6132ec3f7f1278108b856"
Oct 31 01:44:44.481218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.481199    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"
Oct 31 01:44:44.481337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.481232    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d" event={"ID":"99dd44dc-6599-4ec1-812d-d8e4b7144e09","Type":"ContainerDied","Data":"497166d8a65e58499596e7af8e1aa01977ba1487fcd487f4d9468fa7e2dbf6ab"}
Oct 31 01:44:44.481337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.481277    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="497166d8a65e58499596e7af8e1aa01977ba1487fcd487f4d9468fa7e2dbf6ab"
Oct 31 01:44:44.621872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.621838    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"]
Oct 31 01:44:44.622811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.622791    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d02964b6-a6cc-495d-98ee-9b8bff09cadc" containerName="revoke-keys"
Oct 31 01:44:44.622811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.622811    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02964b6-a6cc-495d-98ee-9b8bff09cadc" containerName="revoke-keys"
Oct 31 01:44:44.623016 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.622856    5293 cpu_manager.go:414] "containerMap: removing container" podUID="99dd44dc-6599-4ec1-812d-d8e4b7144e09" containerName="revoke-keys"
Oct 31 01:44:44.623016 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.622864    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99dd44dc-6599-4ec1-812d-d8e4b7144e09" containerName="revoke-keys"
Oct 31 01:44:44.623125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.623106    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02964b6-a6cc-495d-98ee-9b8bff09cadc" containerName="revoke-keys"
Oct 31 01:44:44.623161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.623126    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="99dd44dc-6599-4ec1-812d-d8e4b7144e09" containerName="revoke-keys"
Oct 31 01:44:44.627496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.627478    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.631569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.631540    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:44.631966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.631687    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:44.631966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.631740    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:44:44.631966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.631825    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:44:44.634710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.634688    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"]
Oct 31 01:44:44.817063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.808455    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6jwx\" (UniqueName: \"kubernetes.io/projected/99935a1b-b965-4ce1-999b-b414ec6ecae3-kube-api-access-k6jwx\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.817063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.808590    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.817063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.808763    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.909799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.909762    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.909961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.909865    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6jwx\" (UniqueName: \"kubernetes.io/projected/99935a1b-b965-4ce1-999b-b414ec6ecae3-kube-api-access-k6jwx\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.910035 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.909976    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.910982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.910956    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.912514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.912492    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.921494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.921465    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6jwx\" (UniqueName: \"kubernetes.io/projected/99935a1b-b965-4ce1-999b-b414ec6ecae3-kube-api-access-k6jwx\") pod \"maas-api-revoke-sub-a1d037e2-kn5jx\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:44.953148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:44.953128    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:45.246129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:45.246099    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"]
Oct 31 01:44:45.246853 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:44:45.246827    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99935a1b_b965_4ce1_999b_b414ec6ecae3.slice/crio-e0b0aff939be7ed7d562b0682528edf22f73c135c40bd272e2409929dde3c772 WatchSource:0}: Error finding container e0b0aff939be7ed7d562b0682528edf22f73c135c40bd272e2409929dde3c772: Status 404 returned error can't find the container with id e0b0aff939be7ed7d562b0682528edf22f73c135c40bd272e2409929dde3c772
Oct 31 01:44:45.509170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:45.508219    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx" event={"ID":"99935a1b-b965-4ce1-999b-b414ec6ecae3","Type":"ContainerStarted","Data":"e0b0aff939be7ed7d562b0682528edf22f73c135c40bd272e2409929dde3c772"}
Oct 31 01:44:46.519817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:46.519783    5293 generic.go:356] "Generic (PLEG): container finished" podID="99935a1b-b965-4ce1-999b-b414ec6ecae3" containerID="4a8cd4b1bf37996ac583186eebb2126bb5fafb1ef03b5331f0d9a3ab121c1e34" exitCode=0
Oct 31 01:44:46.520263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:46.519865    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx" event={"ID":"99935a1b-b965-4ce1-999b-b414ec6ecae3","Type":"ContainerDied","Data":"4a8cd4b1bf37996ac583186eebb2126bb5fafb1ef03b5331f0d9a3ab121c1e34"}
Oct 31 01:44:47.697485 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.697455    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:47.856081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.855949    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-openshift-ca\") pod \"99935a1b-b965-4ce1-999b-b414ec6ecae3\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") "
Oct 31 01:44:47.856322 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.856296    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-xks-ca\") pod \"99935a1b-b965-4ce1-999b-b414ec6ecae3\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") "
Oct 31 01:44:47.856449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.856433    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99935a1b-b965-4ce1-999b-b414ec6ecae3-kube-api-access-k6jwx\" (UniqueName: \"kubernetes.io/projected/99935a1b-b965-4ce1-999b-b414ec6ecae3-kube-api-access-k6jwx\") pod \"99935a1b-b965-4ce1-999b-b414ec6ecae3\" (UID: \"99935a1b-b965-4ce1-999b-b414ec6ecae3\") "
Oct 31 01:44:47.857524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.857472    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-openshift-ca" pod "99935a1b-b965-4ce1-999b-b414ec6ecae3" (UID: "99935a1b-b965-4ce1-999b-b414ec6ecae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:47.857714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.857692    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:47.859680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.859659    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-xks-ca" pod "99935a1b-b965-4ce1-999b-b414ec6ecae3" (UID: "99935a1b-b965-4ce1-999b-b414ec6ecae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:47.860053 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.860016    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99935a1b-b965-4ce1-999b-b414ec6ecae3-kube-api-access-k6jwx" pod "99935a1b-b965-4ce1-999b-b414ec6ecae3" (UID: "99935a1b-b965-4ce1-999b-b414ec6ecae3"). InnerVolumeSpecName "kube-api-access-k6jwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:47.959282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.959248    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99935a1b-b965-4ce1-999b-b414ec6ecae3-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:47.959282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:47.959282    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6jwx\" (UniqueName: \"kubernetes.io/projected/99935a1b-b965-4ce1-999b-b414ec6ecae3-kube-api-access-k6jwx\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:48.538391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:48.538306    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"
Oct 31 01:44:48.538391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:48.538324    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx" event={"ID":"99935a1b-b965-4ce1-999b-b414ec6ecae3","Type":"ContainerDied","Data":"e0b0aff939be7ed7d562b0682528edf22f73c135c40bd272e2409929dde3c772"}
Oct 31 01:44:48.538565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:48.538391    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0b0aff939be7ed7d562b0682528edf22f73c135c40bd272e2409929dde3c772"
Oct 31 01:44:50.076386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.076351    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:50.089150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.089128    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:50.196624 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.196593    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:44:50.564841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.564807    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:50.614238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.614206    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"]
Oct 31 01:44:50.615776 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.615756    5293 cpu_manager.go:414] "containerMap: removing container" podUID="99935a1b-b965-4ce1-999b-b414ec6ecae3" containerName="revoke-keys"
Oct 31 01:44:50.615948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.615934    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99935a1b-b965-4ce1-999b-b414ec6ecae3" containerName="revoke-keys"
Oct 31 01:44:50.616244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.616229    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="99935a1b-b965-4ce1-999b-b414ec6ecae3" containerName="revoke-keys"
Oct 31 01:44:50.640878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.640850    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"]
Oct 31 01:44:50.641027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.640945    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.645238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.645212    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:50.645735 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.645713    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:50.645818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.645771    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:44:50.645967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.645946    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:44:50.816131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.811902    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b23f00-d945-478d-953d-47498e96df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.816131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.812025    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlxnf\" (UniqueName: \"kubernetes.io/projected/c5b23f00-d945-478d-953d-47498e96df27-kube-api-access-rlxnf\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.816131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.812158    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b23f00-d945-478d-953d-47498e96df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.920062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.914455    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b23f00-d945-478d-953d-47498e96df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.920062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.916859    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b23f00-d945-478d-953d-47498e96df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.920062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.917116    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b23f00-d945-478d-953d-47498e96df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.920062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.917199    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlxnf\" (UniqueName: \"kubernetes.io/projected/c5b23f00-d945-478d-953d-47498e96df27-kube-api-access-rlxnf\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.943060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.939883    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlxnf\" (UniqueName: \"kubernetes.io/projected/c5b23f00-d945-478d-953d-47498e96df27-kube-api-access-rlxnf\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.947061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.944807    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b23f00-d945-478d-953d-47498e96df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd2e958-9mqnv\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:50.970660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:50.962253    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:51.626067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:51.619324    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"]
Oct 31 01:44:51.639079 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:44:51.635627    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b23f00_d945_478d_953d_47498e96df27.slice/crio-94679ec47aab5abbd707210076cffe33202ff8638c5b67670841c8653b790268 WatchSource:0}: Error finding container 94679ec47aab5abbd707210076cffe33202ff8638c5b67670841c8653b790268: Status 404 returned error can't find the container with id 94679ec47aab5abbd707210076cffe33202ff8638c5b67670841c8653b790268
Oct 31 01:44:52.381125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.373116    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.381125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.373420    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:44:52.385156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.385122    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:44:52.390601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.390555    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 31 01:44:52.390827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.390805    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:44:52.391257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.391234    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:44:52.392004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.391984    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:52.392362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.392343    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:44:52.393705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.393683    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:52.545031 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.544677    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.562009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.561636    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.632261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.632152    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.665078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.664741    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv" event={"ID":"c5b23f00-d945-478d-953d-47498e96df27","Type":"ContainerStarted","Data":"e38c97cbd0894e18b19176d5d358108e0ddbe1506b09c183862dcb015ac6b3b3"}
Oct 31 01:44:52.665078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.664809    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv" event={"ID":"c5b23f00-d945-478d-953d-47498e96df27","Type":"ContainerStarted","Data":"94679ec47aab5abbd707210076cffe33202ff8638c5b67670841c8653b790268"}
Oct 31 01:44:52.726070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.706778    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv" podStartSLOduration=2.70675449 podStartE2EDuration="2.70675449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:52.695299662 +0000 UTC m=+1450.643948056" watchObservedRunningTime="2026-10-31 01:44:52.70675449 +0000 UTC m=+1450.655402886"
Oct 31 01:44:52.801154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:52.801113    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 31 01:44:53.393086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.391513    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.393086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.392268    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.393086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.392832    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:53.393086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.392886    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:44:53.419069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.408750    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.419069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.409091    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.419069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.409350    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.419069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.409604    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:44:53.419069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.409656    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:44:53.419069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.409687    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:44:53.419069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.411201    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.422105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.420010    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:44:53.430516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.395825    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.430516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.424184    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:44:53.585121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.577847    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.609479 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.595142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.624069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.619146    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:53.624069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.619218    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:44:53.647071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.638222    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.647071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.638617    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:44:53.794102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:53.785861    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:54.398063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.397933    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:54.450419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.418237    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.450419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.419708    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.564076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.558150    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:54.740519 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.740422    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.929094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.755121    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.955078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.755154    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:44:55.002162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.755169    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:44:55.028005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.755181    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:55.029318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.786398    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:44:55.030476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.786423    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:44:55.056071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.788303    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded"
Oct 31 01:44:55.056302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:54.903330    5293 generic.go:356] "Generic (PLEG): container finished" podID="c5b23f00-d945-478d-953d-47498e96df27" containerID="e38c97cbd0894e18b19176d5d358108e0ddbe1506b09c183862dcb015ac6b3b3" exitCode=0
Oct 31 01:44:55.232064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.216702    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv" event={"ID":"c5b23f00-d945-478d-953d-47498e96df27","Type":"ContainerDied","Data":"e38c97cbd0894e18b19176d5d358108e0ddbe1506b09c183862dcb015ac6b3b3"}
Oct 31 01:44:55.480125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.376156    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:55.480125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.410402    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:55.621063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.588611    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:55.756744 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.596706    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:55.758755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.657465    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:44:55.863080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.863029    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:44:55.887188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.875562    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:44:55.888252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.888223    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"]
Oct 31 01:44:55.916380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.909792    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:55.954647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:55.954604    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.099129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.089551    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.125708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.125667    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.126792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.126194    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.127208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.127186    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.128351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.128326    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.150834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.150806    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:44:56.151163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.151143    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:44:56.151333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.151313    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"]
Oct 31 01:44:56.151538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.151526    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.160541 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.160512    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ddaae62-bb16-4c30-926b-57812db835fb-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.170160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.170123    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqzb\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-kube-api-access-lsqzb\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.170474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.170455    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.170623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.170600    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.170748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.170734    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.170863 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.170850    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.171053 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.171023    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.171163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.171150    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ddaae62-bb16-4c30-926b-57812db835fb-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.171306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.171290    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.253407 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.252477    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:44:56.253407 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.252529    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"]
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.272989    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ddaae62-bb16-4c30-926b-57812db835fb-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.273033    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqzb\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-kube-api-access-lsqzb\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.273138    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.273208    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.273240    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.273288    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.273364    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.273399    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ddaae62-bb16-4c30-926b-57812db835fb-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.273959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.273435    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.294182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.286469    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.294552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.294531    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:44:56.294702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.294689    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"]
Oct 31 01:44:56.298183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.295641    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.319600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.307451    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.319600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.307719    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.319600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.307970    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.319600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.309402    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ddaae62-bb16-4c30-926b-57812db835fb-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.348013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.324880    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:44:56.356666 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.349599    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ddaae62-bb16-4c30-926b-57812db835fb-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.356666 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.349765    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:44:56.359217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.359197    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:44:56.383165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383074    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.383165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383131    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.383165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383162    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.383440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383214    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.383440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383243    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxmc\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-kube-api-access-llxmc\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.383440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383418    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.383590 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383493    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.383590 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383523    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.383754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.383701    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.476376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.454933    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqzb\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-kube-api-access-lsqzb\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.567867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.519159    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.576062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.573203    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:44:56.593279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.593235    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:44:56.610478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.610215    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.610915 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.610717    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.612226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.611555    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.612861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.612841    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.613676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.613657    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.639159    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.639233    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.639270    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxmc\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-kube-api-access-llxmc\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.623688    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.629892    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.632820    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.639871    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.617308    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.640673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.617166    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.644264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.618449    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.644264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.641501    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.645605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.645577    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.656506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.656463    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.665112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.663814    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.669926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.669896    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxmc\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-kube-api-access-llxmc\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:56.686988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.686906    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:56.741028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.740992    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b23f00-d945-478d-953d-47498e96df27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b23f00-d945-478d-953d-47498e96df27-maas-api-openshift-ca\") pod \"c5b23f00-d945-478d-953d-47498e96df27\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") "
Oct 31 01:44:56.741236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.741165    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5b23f00-d945-478d-953d-47498e96df27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b23f00-d945-478d-953d-47498e96df27-maas-api-xks-ca\") pod \"c5b23f00-d945-478d-953d-47498e96df27\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") "
Oct 31 01:44:56.741454 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.741427    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b23f00-d945-478d-953d-47498e96df27-maas-api-openshift-ca" pod "c5b23f00-d945-478d-953d-47498e96df27" (UID: "c5b23f00-d945-478d-953d-47498e96df27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:56.741525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.741469    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b23f00-d945-478d-953d-47498e96df27-kube-api-access-rlxnf\" (UniqueName: \"kubernetes.io/projected/c5b23f00-d945-478d-953d-47498e96df27-kube-api-access-rlxnf\") pod \"c5b23f00-d945-478d-953d-47498e96df27\" (UID: \"c5b23f00-d945-478d-953d-47498e96df27\") "
Oct 31 01:44:56.749326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.744105    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b23f00-d945-478d-953d-47498e96df27-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:56.749326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.746141    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b23f00-d945-478d-953d-47498e96df27-kube-api-access-rlxnf" pod "c5b23f00-d945-478d-953d-47498e96df27" (UID: "c5b23f00-d945-478d-953d-47498e96df27"). InnerVolumeSpecName "kube-api-access-rlxnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:56.750433 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.750403    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b23f00-d945-478d-953d-47498e96df27-maas-api-xks-ca" pod "c5b23f00-d945-478d-953d-47498e96df27" (UID: "c5b23f00-d945-478d-953d-47498e96df27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:56.824586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.824557    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:56.845072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.845012    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b23f00-d945-478d-953d-47498e96df27-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:56.845176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.845091    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlxnf\" (UniqueName: \"kubernetes.io/projected/c5b23f00-d945-478d-953d-47498e96df27-kube-api-access-rlxnf\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:56.961076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:56.960375    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:57.131697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.131670    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"]
Oct 31 01:44:57.132729 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:44:57.132699    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddaae62_bb16_4c30_926b_57812db835fb.slice/crio-e900aeae9d818b052107b301c7a350e12f1bbc479a5a4b18ec6a9a4f79bab3ee WatchSource:0}: Error finding container e900aeae9d818b052107b301c7a350e12f1bbc479a5a4b18ec6a9a4f79bab3ee: Status 404 returned error can't find the container with id e900aeae9d818b052107b301c7a350e12f1bbc479a5a4b18ec6a9a4f79bab3ee
Oct 31 01:44:57.135707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.135670    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:57.135783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.135745    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:57.135819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.135786    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:57.170595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.170563    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" event={"ID":"7ddaae62-bb16-4c30-926b-57812db835fb","Type":"ContainerStarted","Data":"e900aeae9d818b052107b301c7a350e12f1bbc479a5a4b18ec6a9a4f79bab3ee"}
Oct 31 01:44:57.172660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.172633    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"
Oct 31 01:44:57.173292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.172786    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv" event={"ID":"c5b23f00-d945-478d-953d-47498e96df27","Type":"ContainerDied","Data":"94679ec47aab5abbd707210076cffe33202ff8638c5b67670841c8653b790268"}
Oct 31 01:44:57.173292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.172837    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94679ec47aab5abbd707210076cffe33202ff8638c5b67670841c8653b790268"
Oct 31 01:44:57.259614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.259478    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"]
Oct 31 01:44:57.334075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.330224    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:57.334075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.330345    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:57.334075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:57.330447    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:44:58.331066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.331011    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" event={"ID":"3fe2189d-b5f0-4482-a55f-bd6056e83e52","Type":"ContainerStarted","Data":"bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9"}
Oct 31 01:44:58.331673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.331650    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" event={"ID":"3fe2189d-b5f0-4482-a55f-bd6056e83e52","Type":"ContainerStarted","Data":"f8c8310a68f133bb5bc172ac55b378fe0cbfe730a3f38cdaf3bc95368a5f4cbc"}
Oct 31 01:44:58.335593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.335566    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" event={"ID":"7ddaae62-bb16-4c30-926b-57812db835fb","Type":"ContainerStarted","Data":"e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391"}
Oct 31 01:44:58.385808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.385739    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podStartSLOduration=4.385718789 podStartE2EDuration="4.385718789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:58.370290481 +0000 UTC m=+1456.318938873" watchObservedRunningTime="2026-10-31 01:44:58.385718789 +0000 UTC m=+1456.334367182"
Oct 31 01:44:58.420064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.419817    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podStartSLOduration=4.419795588 podStartE2EDuration="4.419795588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:58.401523248 +0000 UTC m=+1456.350171640" watchObservedRunningTime="2026-10-31 01:44:58.419795588 +0000 UTC m=+1456.368443981"
Oct 31 01:44:58.828257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.828202    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:44:58.830242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.830213    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:44:58.830506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.830464    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 01:44:58.966749 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.966147    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:44:58.967141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.967104    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:44:58.967316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:58.967287    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:44:59.826065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:59.825494    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:44:59.826065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:59.825577    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 01:44:59.961634 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:59.961599    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:44:59.961935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:44:59.961891    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:45:00.280855 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.280828    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"]
Oct 31 01:45:00.281726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.281704    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c5b23f00-d945-478d-953d-47498e96df27" containerName="revoke-keys"
Oct 31 01:45:00.281845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.281834    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b23f00-d945-478d-953d-47498e96df27" containerName="revoke-keys"
Oct 31 01:45:00.282091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.282074    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b23f00-d945-478d-953d-47498e96df27" containerName="revoke-keys"
Oct 31 01:45:00.285240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.285220    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"]
Oct 31 01:45:00.285357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.285337    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:00.289376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.289347    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"]
Oct 31 01:45:00.289558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.289540    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:45:00.289660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.289454    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:00.289660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.289646    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:45:00.291859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.291836    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:45:00.294414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.294398    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"]
Oct 31 01:45:00.294813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.294798    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"]
Oct 31 01:45:00.294936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.294924    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"]
Oct 31 01:45:00.328091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.294501    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:00.469414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.462154    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"]
Oct 31 01:45:00.469414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.462206    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"]
Oct 31 01:45:00.469414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.462326    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:00.469414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.464615    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc25a14-82a6-42cb-a31d-1f366849dfd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890185-9vkwz\" (UID: \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:00.469414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.464684    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq\" (UID: \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:00.469414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.464731    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bq7q\" (UniqueName: \"kubernetes.io/projected/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-kube-api-access-8bq7q\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq\" (UID: \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:00.469414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.464861    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpg77\" (UniqueName: \"kubernetes.io/projected/1cc25a14-82a6-42cb-a31d-1f366849dfd6-kube-api-access-bpg77\") pod \"maas-api-key-cleanup-29890185-9vkwz\" (UID: \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:00.470237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.464918    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrr5c\" (UniqueName: \"kubernetes.io/projected/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-kube-api-access-nrr5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m\" (UID: \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:00.470375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.470354    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m\" (UID: \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:00.495651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.495630    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"]
Oct 31 01:45:00.564117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.563254    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"]
Oct 31 01:45:00.564117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.563390    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.574453    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrr5c\" (UniqueName: \"kubernetes.io/projected/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-kube-api-access-nrr5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m\" (UID: \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.574515    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m\" (UID: \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.574672    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e41dcb5-d668-4218-814b-a711b310be31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk\" (UID: \"6e41dcb5-d668-4218-814b-a711b310be31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.574718    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrmcn\" (UniqueName: \"kubernetes.io/projected/6e41dcb5-d668-4218-814b-a711b310be31-kube-api-access-vrmcn\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk\" (UID: \"6e41dcb5-d668-4218-814b-a711b310be31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.574771    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc25a14-82a6-42cb-a31d-1f366849dfd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890185-9vkwz\" (UID: \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.574828    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq\" (UID: \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.574863    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bq7q\" (UniqueName: \"kubernetes.io/projected/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-kube-api-access-8bq7q\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq\" (UID: \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.574998    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpg77\" (UniqueName: \"kubernetes.io/projected/1cc25a14-82a6-42cb-a31d-1f366849dfd6-kube-api-access-bpg77\") pod \"maas-api-key-cleanup-29890185-9vkwz\" (UID: \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.576195    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m\" (UID: \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:00.577274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.576813    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc25a14-82a6-42cb-a31d-1f366849dfd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890185-9vkwz\" (UID: \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:00.577897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.577442    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:45:00.577897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.577457    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq\" (UID: \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:00.587934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.587410    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrr5c\" (UniqueName: \"kubernetes.io/projected/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-kube-api-access-nrr5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m\" (UID: \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:00.588083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.588065    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpg77\" (UniqueName: \"kubernetes.io/projected/1cc25a14-82a6-42cb-a31d-1f366849dfd6-kube-api-access-bpg77\") pod \"maas-api-key-cleanup-29890185-9vkwz\" (UID: \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:00.589548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.589525    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bq7q\" (UniqueName: \"kubernetes.io/projected/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-kube-api-access-8bq7q\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq\" (UID: \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:00.676150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.676117    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e41dcb5-d668-4218-814b-a711b310be31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk\" (UID: \"6e41dcb5-d668-4218-814b-a711b310be31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:00.676359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.676179    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrmcn\" (UniqueName: \"kubernetes.io/projected/6e41dcb5-d668-4218-814b-a711b310be31-kube-api-access-vrmcn\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk\" (UID: \"6e41dcb5-d668-4218-814b-a711b310be31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:00.676359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.676334    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhzp7\" (UniqueName: \"kubernetes.io/projected/863a76e1-6afc-4434-8b77-4f46f4b86be5-kube-api-access-fhzp7\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.676496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.676461    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.676559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.676516    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.676738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.676715    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e41dcb5-d668-4218-814b-a711b310be31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk\" (UID: \"6e41dcb5-d668-4218-814b-a711b310be31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:00.688090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.688063    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrmcn\" (UniqueName: \"kubernetes.io/projected/6e41dcb5-d668-4218-814b-a711b310be31-kube-api-access-vrmcn\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk\" (UID: \"6e41dcb5-d668-4218-814b-a711b310be31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:00.855115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.827814    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzp7\" (UniqueName: \"kubernetes.io/projected/863a76e1-6afc-4434-8b77-4f46f4b86be5-kube-api-access-fhzp7\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.855115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.827891    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.955064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.866886    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:00.955064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.867741    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:00.955064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.869493    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:00.955064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.869567    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:45:00.955064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.869620    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 01:45:00.955064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.869580    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:00.955064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.899965    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.976300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.969532    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.976300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.971884    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:45:00.976300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.971931    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:45:00.976300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.975880    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzp7\" (UniqueName: \"kubernetes.io/projected/863a76e1-6afc-4434-8b77-4f46f4b86be5-kube-api-access-fhzp7\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:00.977360 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:00.977208    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e20db3-m4qkx\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:01.191063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:01.189704    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:01.863161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:01.862578    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:45:01.863161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:01.862650    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 01:45:01.981032 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:01.980994    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:45:01.981251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:01.981062    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:45:02.094494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.091989    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"]
Oct 31 01:45:02.125108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.096580    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"]
Oct 31 01:45:02.148066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.141839    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"]
Oct 31 01:45:02.192735 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.192709    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"]
Oct 31 01:45:02.265640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.264696    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"]
Oct 31 01:45:02.274917 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:45:02.274793    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863a76e1_6afc_4434_8b77_4f46f4b86be5.slice/crio-9c724735239bf9ad4389097f989967601f08f1c0ae50bf08e48b8a38aedd0a92 WatchSource:0}: Error finding container 9c724735239bf9ad4389097f989967601f08f1c0ae50bf08e48b8a38aedd0a92: Status 404 returned error can't find the container with id 9c724735239bf9ad4389097f989967601f08f1c0ae50bf08e48b8a38aedd0a92
Oct 31 01:45:02.683068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.669349    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" event={"ID":"6e41dcb5-d668-4218-814b-a711b310be31","Type":"ContainerStarted","Data":"807f1f2f1ba3105740e340ecce6c3602fa691ba1568c58c009f927cc527cf772"}
Oct 31 01:45:02.683068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.675331    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" event={"ID":"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47","Type":"ContainerStarted","Data":"fc8e7aa38946ba56136b5de87b5aac5851285558695dbb67f9191af58ad4a736"}
Oct 31 01:45:02.686926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.686891    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" event={"ID":"08b4a678-f59c-4a94-a078-7ed9cb89b8b9","Type":"ContainerStarted","Data":"c3dbfdb1466d799da61d95a4f07b28179bd89d1815cf760c73f04ea6e1ef21d0"}
Oct 31 01:45:02.688563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.688529    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" event={"ID":"1cc25a14-82a6-42cb-a31d-1f366849dfd6","Type":"ContainerStarted","Data":"d40186d1a44a24494854c056ad64aafb261005a95d1ead89a57c9f9c88cfd6e2"}
Oct 31 01:45:02.690192 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.690169    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx" event={"ID":"863a76e1-6afc-4434-8b77-4f46f4b86be5","Type":"ContainerStarted","Data":"9c724735239bf9ad4389097f989967601f08f1c0ae50bf08e48b8a38aedd0a92"}
Oct 31 01:45:02.864088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.853379    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:45:02.864088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.853446    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 01:45:02.965930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.965430    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:45:02.965930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:02.965490    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:45:03.705064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.703425    5293 generic.go:356] "Generic (PLEG): container finished" podID="863a76e1-6afc-4434-8b77-4f46f4b86be5" containerID="7fbb9f24b838b9271bbb10f567afc4908b0cf7483c163ac96d44ec0c073040c8" exitCode=0
Oct 31 01:45:03.705064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.703632    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx" event={"ID":"863a76e1-6afc-4434-8b77-4f46f4b86be5","Type":"ContainerDied","Data":"7fbb9f24b838b9271bbb10f567afc4908b0cf7483c163ac96d44ec0c073040c8"}
Oct 31 01:45:03.720944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.720917    5293 generic.go:356] "Generic (PLEG): container finished" podID="6e41dcb5-d668-4218-814b-a711b310be31" containerID="73e88aa181e9d13458b39ad001177ab4213832bc79bcef7bb7f2b95cac7e5ff7" exitCode=60
Oct 31 01:45:03.721216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.721196    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" event={"ID":"6e41dcb5-d668-4218-814b-a711b310be31","Type":"ContainerDied","Data":"73e88aa181e9d13458b39ad001177ab4213832bc79bcef7bb7f2b95cac7e5ff7"}
Oct 31 01:45:03.721753 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.721735    5293 scope.go:122] "RemoveContainer" containerID="73e88aa181e9d13458b39ad001177ab4213832bc79bcef7bb7f2b95cac7e5ff7"
Oct 31 01:45:03.731780 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.731643    5293 scope.go:122] "RemoveContainer" containerID="58200303a4512c9e12742477ba9e0e3e8fdc849c6271cb99a4346152cd80ebbc"
Oct 31 01:45:03.732124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.732103    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" event={"ID":"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47","Type":"ContainerDied","Data":"58200303a4512c9e12742477ba9e0e3e8fdc849c6271cb99a4346152cd80ebbc"}
Oct 31 01:45:03.732269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.732140    5293 generic.go:356] "Generic (PLEG): container finished" podID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerID="58200303a4512c9e12742477ba9e0e3e8fdc849c6271cb99a4346152cd80ebbc" exitCode=60
Oct 31 01:45:03.742299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.742273    5293 generic.go:356] "Generic (PLEG): container finished" podID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerID="c0e8f16e9295c85669c45ccf481f54d09a7c00f7c4ba36160d494705ff02d046" exitCode=60
Oct 31 01:45:03.742873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.742644    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" event={"ID":"08b4a678-f59c-4a94-a078-7ed9cb89b8b9","Type":"ContainerDied","Data":"c0e8f16e9295c85669c45ccf481f54d09a7c00f7c4ba36160d494705ff02d046"}
Oct 31 01:45:03.743027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.742772    5293 scope.go:122] "RemoveContainer" containerID="c0e8f16e9295c85669c45ccf481f54d09a7c00f7c4ba36160d494705ff02d046"
Oct 31 01:45:03.749933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.746231    5293 generic.go:356] "Generic (PLEG): container finished" podID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerID="1c0c151f56ba01e5753e7539d59ac06693f2bc6636662a41ae2975f4151a7bc7" exitCode=60
Oct 31 01:45:03.749933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.746278    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" event={"ID":"1cc25a14-82a6-42cb-a31d-1f366849dfd6","Type":"ContainerDied","Data":"1c0c151f56ba01e5753e7539d59ac06693f2bc6636662a41ae2975f4151a7bc7"}
Oct 31 01:45:03.749933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.746603    5293 scope.go:122] "RemoveContainer" containerID="1c0c151f56ba01e5753e7539d59ac06693f2bc6636662a41ae2975f4151a7bc7"
Oct 31 01:45:03.826058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.826015    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:45:03.826256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.826231    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 01:45:03.961291 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.961205    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:45:03.961291 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:03.961272    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:45:04.757204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:04.757165    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" event={"ID":"6e41dcb5-d668-4218-814b-a711b310be31","Type":"ContainerStarted","Data":"7b4f01951655fb43178829d894c900ea0596f3f54328ecdaf5332711db0fc725"}
Oct 31 01:45:04.760027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:04.759996    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" event={"ID":"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47","Type":"ContainerStarted","Data":"18d5aface07f1a99e4d65952b34675bfa01345a86f44397a642243deb2ebf1ac"}
Oct 31 01:45:04.769718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:04.764097    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" event={"ID":"08b4a678-f59c-4a94-a078-7ed9cb89b8b9","Type":"ContainerStarted","Data":"73acb1ac5a6c646d2f09ffcd5a657f3c65ed415658bfa613d18a67b7ee86fd0c"}
Oct 31 01:45:04.769718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:04.767339    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" event={"ID":"1cc25a14-82a6-42cb-a31d-1f366849dfd6","Type":"ContainerStarted","Data":"a5e0e869b93488464beeb02c3c9b603c7f6664353d7420b0ac5060c2aa52620f"}
Oct 31 01:45:04.782065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:04.782008    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" podStartSLOduration=4.7819961079999995 podStartE2EDuration="4.781996108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:04.77481107 +0000 UTC m=+1462.723459464" watchObservedRunningTime="2026-10-31 01:45:04.781996108 +0000 UTC m=+1462.730644524"
Oct 31 01:45:04.834630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:04.834537    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" podStartSLOduration=4.834515155 podStartE2EDuration="4.834515155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:04.795107241 +0000 UTC m=+1462.743755635" watchObservedRunningTime="2026-10-31 01:45:04.834515155 +0000 UTC m=+1462.783163540"
Oct 31 01:45:04.889069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:04.887382    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" podStartSLOduration=4.8873545400000005 podStartE2EDuration="4.88735454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:04.845123671 +0000 UTC m=+1462.793772064" watchObservedRunningTime="2026-10-31 01:45:04.88735454 +0000 UTC m=+1462.836002933"
Oct 31 01:45:04.959689 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:04.952604    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" podStartSLOduration=4.952579125 podStartE2EDuration="4.952579125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:04.865998107 +0000 UTC m=+1462.814646502" watchObservedRunningTime="2026-10-31 01:45:04.952579125 +0000 UTC m=+1462.901227577"
Oct 31 01:45:05.215595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.215241    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:05.329688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.329607    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:45:05.331532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.331474    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:45:05.333767 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.333719    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 01:45:05.361425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.361394    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863a76e1-6afc-4434-8b77-4f46f4b86be5-kube-api-access-fhzp7\" (UniqueName: \"kubernetes.io/projected/863a76e1-6afc-4434-8b77-4f46f4b86be5-kube-api-access-fhzp7\") pod \"863a76e1-6afc-4434-8b77-4f46f4b86be5\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") "
Oct 31 01:45:05.361770 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.361736    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-openshift-ca\") pod \"863a76e1-6afc-4434-8b77-4f46f4b86be5\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") "
Oct 31 01:45:05.362084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.362064    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-xks-ca\") pod \"863a76e1-6afc-4434-8b77-4f46f4b86be5\" (UID: \"863a76e1-6afc-4434-8b77-4f46f4b86be5\") "
Oct 31 01:45:05.366618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.366577    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-openshift-ca" pod "863a76e1-6afc-4434-8b77-4f46f4b86be5" (UID: "863a76e1-6afc-4434-8b77-4f46f4b86be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:05.375284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.375258    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-xks-ca" pod "863a76e1-6afc-4434-8b77-4f46f4b86be5" (UID: "863a76e1-6afc-4434-8b77-4f46f4b86be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:45:05.383240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.380189    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863a76e1-6afc-4434-8b77-4f46f4b86be5-kube-api-access-fhzp7" pod "863a76e1-6afc-4434-8b77-4f46f4b86be5" (UID: "863a76e1-6afc-4434-8b77-4f46f4b86be5"). InnerVolumeSpecName "kube-api-access-fhzp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:05.478063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.471347    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:05.478063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.471384    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhzp7\" (UniqueName: \"kubernetes.io/projected/863a76e1-6afc-4434-8b77-4f46f4b86be5-kube-api-access-fhzp7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:05.478063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.471402    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863a76e1-6afc-4434-8b77-4f46f4b86be5-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:05.814511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.814467    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx" event={"ID":"863a76e1-6afc-4434-8b77-4f46f4b86be5","Type":"ContainerDied","Data":"9c724735239bf9ad4389097f989967601f08f1c0ae50bf08e48b8a38aedd0a92"}
Oct 31 01:45:05.814834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.814808    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c724735239bf9ad4389097f989967601f08f1c0ae50bf08e48b8a38aedd0a92"
Oct 31 01:45:05.815200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.815176    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"
Oct 31 01:45:05.845063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.841230    5293 generic.go:356] "Generic (PLEG): container finished" podID="6e41dcb5-d668-4218-814b-a711b310be31" containerID="7b4f01951655fb43178829d894c900ea0596f3f54328ecdaf5332711db0fc725" exitCode=60
Oct 31 01:45:05.845362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.845335    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" event={"ID":"6e41dcb5-d668-4218-814b-a711b310be31","Type":"ContainerDied","Data":"7b4f01951655fb43178829d894c900ea0596f3f54328ecdaf5332711db0fc725"}
Oct 31 01:45:05.845497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.845485    5293 scope.go:122] "RemoveContainer" containerID="73e88aa181e9d13458b39ad001177ab4213832bc79bcef7bb7f2b95cac7e5ff7"
Oct 31 01:45:05.858817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:05.846126    5293 scope.go:122] "RemoveContainer" containerID="7b4f01951655fb43178829d894c900ea0596f3f54328ecdaf5332711db0fc725"
Oct 31 01:45:05.859721 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:05.859673    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk_odh-ai-gateway-infra(6e41dcb5-d668-4218-814b-a711b310be31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" podUID="6e41dcb5-d668-4218-814b-a711b310be31"
Oct 31 01:45:06.176071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.156769    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:06.176071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.156835    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:06.176071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.162768    5293 generic.go:356] "Generic (PLEG): container finished" podID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerID="18d5aface07f1a99e4d65952b34675bfa01345a86f44397a642243deb2ebf1ac" exitCode=60
Oct 31 01:45:06.176071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.162866    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" event={"ID":"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47","Type":"ContainerDied","Data":"18d5aface07f1a99e4d65952b34675bfa01345a86f44397a642243deb2ebf1ac"}
Oct 31 01:45:06.176071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.162944    5293 scope.go:122] "RemoveContainer" containerID="58200303a4512c9e12742477ba9e0e3e8fdc849c6271cb99a4346152cd80ebbc"
Oct 31 01:45:06.176071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.163480    5293 scope.go:122] "RemoveContainer" containerID="18d5aface07f1a99e4d65952b34675bfa01345a86f44397a642243deb2ebf1ac"
Oct 31 01:45:06.209130 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:06.177525    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m_odh-ai-gateway-infra(bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47"
Oct 31 01:45:06.386068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.328014    5293 generic.go:356] "Generic (PLEG): container finished" podID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerID="73acb1ac5a6c646d2f09ffcd5a657f3c65ed415658bfa613d18a67b7ee86fd0c" exitCode=60
Oct 31 01:45:06.386068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.328750    5293 scope.go:122] "RemoveContainer" containerID="73acb1ac5a6c646d2f09ffcd5a657f3c65ed415658bfa613d18a67b7ee86fd0c"
Oct 31 01:45:06.386068 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:06.329219    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq_odh-ai-gateway-infra(08b4a678-f59c-4a94-a078-7ed9cb89b8b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9"
Oct 31 01:45:06.386068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.347682    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" event={"ID":"08b4a678-f59c-4a94-a078-7ed9cb89b8b9","Type":"ContainerDied","Data":"73acb1ac5a6c646d2f09ffcd5a657f3c65ed415658bfa613d18a67b7ee86fd0c"}
Oct 31 01:45:06.436779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.436699    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:45:06.438315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.438295    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:45:06.458249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.458222    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:45:06.534183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.530371    5293 scope.go:122] "RemoveContainer" containerID="c0e8f16e9295c85669c45ccf481f54d09a7c00f7c4ba36160d494705ff02d046"
Oct 31 01:45:06.548345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.545891    5293 generic.go:356] "Generic (PLEG): container finished" podID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerID="a5e0e869b93488464beeb02c3c9b603c7f6664353d7420b0ac5060c2aa52620f" exitCode=60
Oct 31 01:45:06.548345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.547162    5293 scope.go:122] "RemoveContainer" containerID="a5e0e869b93488464beeb02c3c9b603c7f6664353d7420b0ac5060c2aa52620f"
Oct 31 01:45:06.548345 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:06.547440    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890185-9vkwz_odh-ai-gateway-infra(1cc25a14-82a6-42cb-a31d-1f366849dfd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6"
Oct 31 01:45:06.548345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.547686    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" event={"ID":"1cc25a14-82a6-42cb-a31d-1f366849dfd6","Type":"ContainerDied","Data":"a5e0e869b93488464beeb02c3c9b603c7f6664353d7420b0ac5060c2aa52620f"}
Oct 31 01:45:06.759526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:06.759495    5293 scope.go:122] "RemoveContainer" containerID="1c0c151f56ba01e5753e7539d59ac06693f2bc6636662a41ae2975f4151a7bc7"
Oct 31 01:45:07.617568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:07.617533    5293 scope.go:122] "RemoveContainer" containerID="7b4f01951655fb43178829d894c900ea0596f3f54328ecdaf5332711db0fc725"
Oct 31 01:45:07.660499 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:07.660461    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk_odh-ai-gateway-infra(6e41dcb5-d668-4218-814b-a711b310be31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" podUID="6e41dcb5-d668-4218-814b-a711b310be31"
Oct 31 01:45:07.687067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:07.668280    5293 scope.go:122] "RemoveContainer" containerID="18d5aface07f1a99e4d65952b34675bfa01345a86f44397a642243deb2ebf1ac"
Oct 31 01:45:07.687067 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:07.668591    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m_odh-ai-gateway-infra(bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47"
Oct 31 01:45:07.756498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:07.756468    5293 scope.go:122] "RemoveContainer" containerID="73acb1ac5a6c646d2f09ffcd5a657f3c65ed415658bfa613d18a67b7ee86fd0c"
Oct 31 01:45:07.757107 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:07.757077    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq_odh-ai-gateway-infra(08b4a678-f59c-4a94-a078-7ed9cb89b8b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9"
Oct 31 01:45:07.759164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:07.759133    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:07.826259 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:07.810258    5293 scope.go:122] "RemoveContainer" containerID="a5e0e869b93488464beeb02c3c9b603c7f6664353d7420b0ac5060c2aa52620f"
Oct 31 01:45:07.837561 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:07.837528    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890185-9vkwz_odh-ai-gateway-infra(1cc25a14-82a6-42cb-a31d-1f366849dfd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6"
Oct 31 01:45:08.042517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.041385    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.042517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.041816    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:45:08.128840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.128793    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.129533 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.129114    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.352221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.352128    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.405187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.405147    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.440066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.431777    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.469147    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:08.477381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.477345    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.607882 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.607777    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.764092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.762134    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.764092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.762920    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:45:08.764092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.763093    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.791086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:08.791024    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.002082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.001324    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.090062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.044406    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.090062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.045249    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:45:09.090062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.045892    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:45:09.100728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.100700    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:45:09.202065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.131873    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.202065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.132570    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:45:09.202065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.133449    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:45:09.202065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.133749    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:45:09.383489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.383388    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:45:09.385180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.385153    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:45:09.436133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.436086    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.474066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.437651    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:45:09.474066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.467274    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.474441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.474420    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:45:09.626067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.610850    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.640674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.640575    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.658078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.653600    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:45:09.773069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.772116    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.773069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.772735    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:45:09.773069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.772797    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:45:09.878235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.874548    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:45:09.878235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:09.874637    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.077123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.075463    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:45:10.077123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.076189    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:45:10.096068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.084512    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.096068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.085159    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:45:10.143130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.118024    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:45:10.220167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.220128    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:45:10.248117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.246203    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:45:10.571078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.406441    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 01:45:10.594299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.408778    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.594851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.490454    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.730068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.655457    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.820067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.762512    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:45:10.844795 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.786226    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.877297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:10.873630    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:45:11.277007 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.276963    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:45:11.337079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.337008    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:11.347597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.347543    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:45:11.384061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.382773    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:45:11.387395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.387361    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:11.397209 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.396839    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:45:11.434130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.398852    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:45:11.557183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.557101    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:45:11.559226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.559208    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:45:11.559440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.559427    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:45:11.596509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.596476    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:11.597081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.596991    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:45:11.604366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.604335    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"]
Oct 31 01:45:11.627805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.627727    5293 cpu_manager.go:414] "containerMap: removing container" podUID="863a76e1-6afc-4434-8b77-4f46f4b86be5" containerName="revoke-keys"
Oct 31 01:45:11.627805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.627767    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863a76e1-6afc-4434-8b77-4f46f4b86be5" containerName="revoke-keys"
Oct 31 01:45:11.628384 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.628153    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="863a76e1-6afc-4434-8b77-4f46f4b86be5" containerName="revoke-keys"
Oct 31 01:45:11.651269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.651240    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"]
Oct 31 01:45:11.651530 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.651512    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:45:11.651883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.651862    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:45:11.652341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.652319    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.778837 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:11.778798    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832142    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832206    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5d2s\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-kube-api-access-c5d2s\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832272    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832336    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832454    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832477    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832569    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832617    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.838307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.832659    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934554 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934521    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934559    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934620    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934663    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934701    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934825    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934867    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5d2s\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-kube-api-access-c5d2s\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934909    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.934956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.934950    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.935219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.935091    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.935305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.935283    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.935372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.935357    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.935427 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.935412    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.935477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.935441    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.937378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.937351    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.937607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.937587    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.949328 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.949300    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5d2s\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-kube-api-access-c5d2s\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.952408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.952382    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:11.970627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:11.970606    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:12.262779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:12.262754    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"]
Oct 31 01:45:12.263270 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:45:12.263242    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46cf4fa_53a4_405b_b669_85f5fce9e3e9.slice/crio-3e1b18a3480b874e653d4d8339610485d42e5f83bebd4f7f7a3fbca81848f417 WatchSource:0}: Error finding container 3e1b18a3480b874e653d4d8339610485d42e5f83bebd4f7f7a3fbca81848f417: Status 404 returned error can't find the container with id 3e1b18a3480b874e653d4d8339610485d42e5f83bebd4f7f7a3fbca81848f417
Oct 31 01:45:12.265764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:12.265729    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:12.265848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:12.265802    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:12.265848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:12.265830    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:12.965745 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:12.965710    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" event={"ID":"d46cf4fa-53a4-405b-b669-85f5fce9e3e9","Type":"ContainerStarted","Data":"a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a"}
Oct 31 01:45:12.966407 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:12.966378    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" event={"ID":"d46cf4fa-53a4-405b-b669-85f5fce9e3e9","Type":"ContainerStarted","Data":"3e1b18a3480b874e653d4d8339610485d42e5f83bebd4f7f7a3fbca81848f417"}
Oct 31 01:45:12.972158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:12.972129    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:13.006827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:13.006765    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podStartSLOduration=3.006742378 podStartE2EDuration="3.006742378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:13.000571557 +0000 UTC m=+1470.949219976" watchObservedRunningTime="2026-10-31 01:45:13.006742378 +0000 UTC m=+1470.955390770"
Oct 31 01:45:13.981069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:13.975763    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:45:13.981069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:13.975824    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:45:14.971252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:14.971216    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:45:14.971408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:14.971272    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:45:15.972074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:15.972006    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:45:15.972562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:15.972117    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:45:16.176012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:16.175553    5293 scope.go:122] "RemoveContainer" containerID="73acb1ac5a6c646d2f09ffcd5a657f3c65ed415658bfa613d18a67b7ee86fd0c"
Oct 31 01:45:16.176012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:16.175852    5293 scope.go:122] "RemoveContainer" containerID="18d5aface07f1a99e4d65952b34675bfa01345a86f44397a642243deb2ebf1ac"
Oct 31 01:45:16.176012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:16.175917    5293 scope.go:122] "RemoveContainer" containerID="a5e0e869b93488464beeb02c3c9b603c7f6664353d7420b0ac5060c2aa52620f"
Oct 31 01:45:16.971536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:16.971502    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:45:16.971700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:16.971553    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:45:17.029806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:17.029766    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" event={"ID":"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47","Type":"ContainerStarted","Data":"0790e45dad5e8f980c23ff0847d11c44b24aee0625f664a1ed7ed152cbe12ec3"}
Oct 31 01:45:17.031737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:17.031715    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" event={"ID":"08b4a678-f59c-4a94-a078-7ed9cb89b8b9","Type":"ContainerStarted","Data":"490a8351e160e7135fabf17d4199c8a0a9beef2521ff1e6e2f0fce00446a9755"}
Oct 31 01:45:17.033927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:17.033881    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" event={"ID":"1cc25a14-82a6-42cb-a31d-1f366849dfd6","Type":"ContainerStarted","Data":"b358d86420a06bfb7c8f43368fe1ff66d4950f663733807d14cb29f59325d6cb"}
Oct 31 01:45:17.166260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:17.166225    5293 scope.go:122] "RemoveContainer" containerID="7b4f01951655fb43178829d894c900ea0596f3f54328ecdaf5332711db0fc725"
Oct 31 01:45:17.972509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:17.972474    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:45:17.972786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:17.972755    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:45:18.095065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.088376    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"]
Oct 31 01:45:18.106068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.102100    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" event={"ID":"6e41dcb5-d668-4218-814b-a711b310be31","Type":"ContainerStarted","Data":"6f8cd57cef767a1892e506809884a14a4c1c07d3a5c23c7fc4f2454204407e20"}
Oct 31 01:45:18.130066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.127210    5293 generic.go:356] "Generic (PLEG): container finished" podID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerID="0790e45dad5e8f980c23ff0847d11c44b24aee0625f664a1ed7ed152cbe12ec3" exitCode=60
Oct 31 01:45:18.130066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.127304    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" event={"ID":"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47","Type":"ContainerDied","Data":"0790e45dad5e8f980c23ff0847d11c44b24aee0625f664a1ed7ed152cbe12ec3"}
Oct 31 01:45:18.130066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.127364    5293 scope.go:122] "RemoveContainer" containerID="18d5aface07f1a99e4d65952b34675bfa01345a86f44397a642243deb2ebf1ac"
Oct 31 01:45:18.134990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.130786    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"]
Oct 31 01:45:18.140406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.135563    5293 generic.go:356] "Generic (PLEG): container finished" podID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerID="490a8351e160e7135fabf17d4199c8a0a9beef2521ff1e6e2f0fce00446a9755" exitCode=60
Oct 31 01:45:18.140406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.139814    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" event={"ID":"08b4a678-f59c-4a94-a078-7ed9cb89b8b9","Type":"ContainerDied","Data":"490a8351e160e7135fabf17d4199c8a0a9beef2521ff1e6e2f0fce00446a9755"}
Oct 31 01:45:18.153336 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.151755    5293 generic.go:356] "Generic (PLEG): container finished" podID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerID="b358d86420a06bfb7c8f43368fe1ff66d4950f663733807d14cb29f59325d6cb" exitCode=60
Oct 31 01:45:18.153336 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.152056    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" event={"ID":"1cc25a14-82a6-42cb-a31d-1f366849dfd6","Type":"ContainerDied","Data":"b358d86420a06bfb7c8f43368fe1ff66d4950f663733807d14cb29f59325d6cb"}
Oct 31 01:45:18.153336 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.152572    5293 scope.go:122] "RemoveContainer" containerID="b358d86420a06bfb7c8f43368fe1ff66d4950f663733807d14cb29f59325d6cb"
Oct 31 01:45:18.153336 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:18.152979    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890185-9vkwz_odh-ai-gateway-infra(1cc25a14-82a6-42cb-a31d-1f366849dfd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6"
Oct 31 01:45:18.179117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.179095    5293 scope.go:122] "RemoveContainer" containerID="73acb1ac5a6c646d2f09ffcd5a657f3c65ed415658bfa613d18a67b7ee86fd0c"
Oct 31 01:45:18.182852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.182822    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"]
Oct 31 01:45:18.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.219162    5293 scope.go:122] "RemoveContainer" containerID="a5e0e869b93488464beeb02c3c9b603c7f6664353d7420b0ac5060c2aa52620f"
Oct 31 01:45:18.483317 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.483294    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:18.644081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.640301    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:18.676097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.674748    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-maas-api-service-ca\") pod \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\" (UID: \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\") "
Oct 31 01:45:18.676097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.675126    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-kube-api-access-nrr5c\" (UniqueName: \"kubernetes.io/projected/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-kube-api-access-nrr5c\") pod \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\" (UID: \"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47\") "
Oct 31 01:45:18.717107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.683949    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-maas-api-service-ca" pod "bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" (UID: "bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:18.717107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.684201    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-kube-api-access-nrr5c" pod "bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" (UID: "bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47"). InnerVolumeSpecName "kube-api-access-nrr5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:18.788988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.781759    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-maas-api-service-ca\") pod \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\" (UID: \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\") "
Oct 31 01:45:18.788988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.782136    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-kube-api-access-8bq7q\" (UniqueName: \"kubernetes.io/projected/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-kube-api-access-8bq7q\") pod \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\" (UID: \"08b4a678-f59c-4a94-a078-7ed9cb89b8b9\") "
Oct 31 01:45:18.788988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.783057    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.788988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.783077    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrr5c\" (UniqueName: \"kubernetes.io/projected/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47-kube-api-access-nrr5c\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.788988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.783790    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-maas-api-service-ca" pod "08b4a678-f59c-4a94-a078-7ed9cb89b8b9" (UID: "08b4a678-f59c-4a94-a078-7ed9cb89b8b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:18.788988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.787288    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-kube-api-access-8bq7q" pod "08b4a678-f59c-4a94-a078-7ed9cb89b8b9" (UID: "08b4a678-f59c-4a94-a078-7ed9cb89b8b9"). InnerVolumeSpecName "kube-api-access-8bq7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:18.885020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.884925    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.885020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.884959    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bq7q\" (UniqueName: \"kubernetes.io/projected/08b4a678-f59c-4a94-a078-7ed9cb89b8b9-kube-api-access-8bq7q\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.989063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.982468    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:45:18.989063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:18.982544    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:45:19.141230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.141143    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"]
Oct 31 01:45:19.161981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.161948    5293 generic.go:356] "Generic (PLEG): container finished" podID="6e41dcb5-d668-4218-814b-a711b310be31" containerID="6f8cd57cef767a1892e506809884a14a4c1c07d3a5c23c7fc4f2454204407e20" exitCode=60
Oct 31 01:45:19.162208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.162031    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" event={"ID":"6e41dcb5-d668-4218-814b-a711b310be31","Type":"ContainerDied","Data":"6f8cd57cef767a1892e506809884a14a4c1c07d3a5c23c7fc4f2454204407e20"}
Oct 31 01:45:19.162208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.162157    5293 scope.go:122] "RemoveContainer" containerID="7b4f01951655fb43178829d894c900ea0596f3f54328ecdaf5332711db0fc725"
Oct 31 01:45:19.164665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.164638    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"
Oct 31 01:45:19.165163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.165138    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m" event={"ID":"bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47","Type":"ContainerDied","Data":"fc8e7aa38946ba56136b5de87b5aac5851285558695dbb67f9191af58ad4a736"}
Oct 31 01:45:19.172054 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.172010    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq" event={"ID":"08b4a678-f59c-4a94-a078-7ed9cb89b8b9","Type":"ContainerDied","Data":"c3dbfdb1466d799da61d95a4f07b28179bd89d1815cf760c73f04ea6e1ef21d0"}
Oct 31 01:45:19.172213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.172195    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"
Oct 31 01:45:19.183725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.183556    5293 scope.go:122] "RemoveContainer" containerID="0790e45dad5e8f980c23ff0847d11c44b24aee0625f664a1ed7ed152cbe12ec3"
Oct 31 01:45:19.201238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.200468    5293 scope.go:122] "RemoveContainer" containerID="490a8351e160e7135fabf17d4199c8a0a9beef2521ff1e6e2f0fce00446a9755"
Oct 31 01:45:19.207737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.207679    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"]
Oct 31 01:45:19.220329 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.220306    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890185-8fp5m"]
Oct 31 01:45:19.252075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.250760    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"]
Oct 31 01:45:19.258018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.257989    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890185-qmlkq"]
Oct 31 01:45:19.520113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.517255    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:19.522625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.520582    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e41dcb5-d668-4218-814b-a711b310be31-kube-api-access-vrmcn\" (UniqueName: \"kubernetes.io/projected/6e41dcb5-d668-4218-814b-a711b310be31-kube-api-access-vrmcn\") pod \"6e41dcb5-d668-4218-814b-a711b310be31\" (UID: \"6e41dcb5-d668-4218-814b-a711b310be31\") "
Oct 31 01:45:19.537894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.537850    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e41dcb5-d668-4218-814b-a711b310be31-kube-api-access-vrmcn" pod "6e41dcb5-d668-4218-814b-a711b310be31" (UID: "6e41dcb5-d668-4218-814b-a711b310be31"). InnerVolumeSpecName "kube-api-access-vrmcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:19.593459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.593422    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:19.636562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.630131    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e41dcb5-d668-4218-814b-a711b310be31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e41dcb5-d668-4218-814b-a711b310be31-maas-api-service-ca\") pod \"6e41dcb5-d668-4218-814b-a711b310be31\" (UID: \"6e41dcb5-d668-4218-814b-a711b310be31\") "
Oct 31 01:45:19.636562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.631110    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrmcn\" (UniqueName: \"kubernetes.io/projected/6e41dcb5-d668-4218-814b-a711b310be31-kube-api-access-vrmcn\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.636562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.635008    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e41dcb5-d668-4218-814b-a711b310be31-maas-api-service-ca" pod "6e41dcb5-d668-4218-814b-a711b310be31" (UID: "6e41dcb5-d668-4218-814b-a711b310be31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:19.747154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.740275    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc25a14-82a6-42cb-a31d-1f366849dfd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc25a14-82a6-42cb-a31d-1f366849dfd6-maas-api-service-ca\") pod \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\" (UID: \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\") "
Oct 31 01:45:19.747154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.740371    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc25a14-82a6-42cb-a31d-1f366849dfd6-kube-api-access-bpg77\" (UniqueName: \"kubernetes.io/projected/1cc25a14-82a6-42cb-a31d-1f366849dfd6-kube-api-access-bpg77\") pod \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\" (UID: \"1cc25a14-82a6-42cb-a31d-1f366849dfd6\") "
Oct 31 01:45:19.747154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.741352    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc25a14-82a6-42cb-a31d-1f366849dfd6-maas-api-service-ca" pod "1cc25a14-82a6-42cb-a31d-1f366849dfd6" (UID: "1cc25a14-82a6-42cb-a31d-1f366849dfd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:19.747154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.741812    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e41dcb5-d668-4218-814b-a711b310be31-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.747154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.741835    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc25a14-82a6-42cb-a31d-1f366849dfd6-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.779274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.779210    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc25a14-82a6-42cb-a31d-1f366849dfd6-kube-api-access-bpg77" pod "1cc25a14-82a6-42cb-a31d-1f366849dfd6" (UID: "1cc25a14-82a6-42cb-a31d-1f366849dfd6"). InnerVolumeSpecName "kube-api-access-bpg77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:19.842889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.842855    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpg77\" (UniqueName: \"kubernetes.io/projected/1cc25a14-82a6-42cb-a31d-1f366849dfd6-kube-api-access-bpg77\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.971613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.971565    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:45:19.971814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:19.971619    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:45:20.175599 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.175570    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" path="/var/lib/kubelet/pods/08b4a678-f59c-4a94-a078-7ed9cb89b8b9/volumes"
Oct 31 01:45:20.176029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.176016    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" path="/var/lib/kubelet/pods/bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47/volumes"
Oct 31 01:45:20.189198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.189179    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"
Oct 31 01:45:20.189316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.189196    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk" event={"ID":"6e41dcb5-d668-4218-814b-a711b310be31","Type":"ContainerDied","Data":"807f1f2f1ba3105740e340ecce6c3602fa691ba1568c58c009f927cc527cf772"}
Oct 31 01:45:20.189316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.189257    5293 scope.go:122] "RemoveContainer" containerID="6f8cd57cef767a1892e506809884a14a4c1c07d3a5c23c7fc4f2454204407e20"
Oct 31 01:45:20.193817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.193802    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"
Oct 31 01:45:20.193917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.193900    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz" event={"ID":"1cc25a14-82a6-42cb-a31d-1f366849dfd6","Type":"ContainerDied","Data":"d40186d1a44a24494854c056ad64aafb261005a95d1ead89a57c9f9c88cfd6e2"}
Oct 31 01:45:20.202525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.202497    5293 scope.go:122] "RemoveContainer" containerID="b358d86420a06bfb7c8f43368fe1ff66d4950f663733807d14cb29f59325d6cb"
Oct 31 01:45:20.216686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.216664    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"]
Oct 31 01:45:20.222218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.222200    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890185-jcxlk"]
Oct 31 01:45:20.233251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.233231    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"]
Oct 31 01:45:20.237309 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.237291    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-9vkwz"]
Oct 31 01:45:20.975306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.975275    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:20.975609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.975578    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:20.976472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:20.976452    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 01:45:21.480248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480213    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"]
Oct 31 01:45:21.480775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480762    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480777    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480791    5293 cpu_manager.go:414] "containerMap: removing container" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480796    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480805    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480811    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480819    5293 cpu_manager.go:414] "containerMap: removing container" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480823    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480831    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.480836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480837    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480845    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480849    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480865    5293 cpu_manager.go:414] "containerMap: removing container" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480870    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480880    5293 cpu_manager.go:414] "containerMap: removing container" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480884    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480908    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480913    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480987    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.480996    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481003    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481008    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481013    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e41dcb5-d668-4218-814b-a711b310be31" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481018    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481024    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481033    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481052    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481058    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:21.481143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.481062    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:21.486021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.486001    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.489406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.489385    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:45:21.489406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.489398    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:45:21.489638 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.489617    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:45:21.489761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.489741    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:45:21.495935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.495909    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"]
Oct 31 01:45:21.576805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.576775    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbh8\" (UniqueName: \"kubernetes.io/projected/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-kube-api-access-4pbh8\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.576964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.576850    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.576964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.576923    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.678153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.678118    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbh8\" (UniqueName: \"kubernetes.io/projected/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-kube-api-access-4pbh8\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.678361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.678173    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.678361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.678203    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.678889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.678868    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.680552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.680534    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.692515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.692490    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbh8\" (UniqueName: \"kubernetes.io/projected/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-kube-api-access-4pbh8\") pod \"maas-api-revoke-sub-bcaa8ee8-969nx\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:21.807411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:21.807303    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:22.118976 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:45:22.118948    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3990a2d_4452_4f9b_9b13_d19e6d9d8e85.slice/crio-5bac80441ea76b42630d8e27f37767132f21b91f3ce4014c963c1b074321db46 WatchSource:0}: Error finding container 5bac80441ea76b42630d8e27f37767132f21b91f3ce4014c963c1b074321db46: Status 404 returned error can't find the container with id 5bac80441ea76b42630d8e27f37767132f21b91f3ce4014c963c1b074321db46
Oct 31 01:45:22.119306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:22.119259    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"]
Oct 31 01:45:22.176153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:22.176127    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" path="/var/lib/kubelet/pods/1cc25a14-82a6-42cb-a31d-1f366849dfd6/volumes"
Oct 31 01:45:22.176606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:22.176566    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e41dcb5-d668-4218-814b-a711b310be31" path="/var/lib/kubelet/pods/6e41dcb5-d668-4218-814b-a711b310be31/volumes"
Oct 31 01:45:22.222028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:22.221988    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx" event={"ID":"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85","Type":"ContainerStarted","Data":"5bac80441ea76b42630d8e27f37767132f21b91f3ce4014c963c1b074321db46"}
Oct 31 01:45:23.110584 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.110549    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"]
Oct 31 01:45:23.111423 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.111402    5293 cpu_manager.go:414] "containerMap: removing container" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:23.111495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.111425    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:23.111495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.111474    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:23.111495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.111480    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc25a14-82a6-42cb-a31d-1f366849dfd6" containerName="cleanup"
Oct 31 01:45:23.111603 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.111502    5293 cpu_manager.go:414] "containerMap: removing container" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:23.111603 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.111509    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfef9cb9-2c8f-4af5-81ea-7ef686ad0a47" containerName="cleanup"
Oct 31 01:45:23.111661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.111651    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b4a678-f59c-4a94-a078-7ed9cb89b8b9" containerName="cleanup"
Oct 31 01:45:23.115843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.115823    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.123596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.123550    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"]
Oct 31 01:45:23.212706 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.212672    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.212867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.212826    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwxw\" (UniqueName: \"kubernetes.io/projected/7e336afe-c771-4703-8519-63dd8c922a24-kube-api-access-ldwxw\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.212867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.212851    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.231331 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.231301    5293 generic.go:356] "Generic (PLEG): container finished" podID="f3990a2d-4452-4f9b-9b13-d19e6d9d8e85" containerID="4cd0361c8bbb98a4907c6ea464ab76e25bb56db44ef0407b16f4048465ffd46e" exitCode=0
Oct 31 01:45:23.231467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.231382    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx" event={"ID":"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85","Type":"ContainerDied","Data":"4cd0361c8bbb98a4907c6ea464ab76e25bb56db44ef0407b16f4048465ffd46e"}
Oct 31 01:45:23.330107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.318401    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.330107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.318492    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwxw\" (UniqueName: \"kubernetes.io/projected/7e336afe-c771-4703-8519-63dd8c922a24-kube-api-access-ldwxw\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.330107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.318520    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.330107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.324235    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.349508 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.344962    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.349508 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.348637    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwxw\" (UniqueName: \"kubernetes.io/projected/7e336afe-c771-4703-8519-63dd8c922a24-kube-api-access-ldwxw\") pod \"maas-api-revoke-sub-a8a43f65-gz9lg\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.434187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.434160    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:23.725495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:23.725466    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"]
Oct 31 01:45:23.727392 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:45:23.727367    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e336afe_c771_4703_8519_63dd8c922a24.slice/crio-c2d649463aa54c44564f301fcd6bf3e195b6724d374c4a2b8c0c97086eed307f WatchSource:0}: Error finding container c2d649463aa54c44564f301fcd6bf3e195b6724d374c4a2b8c0c97086eed307f: Status 404 returned error can't find the container with id c2d649463aa54c44564f301fcd6bf3e195b6724d374c4a2b8c0c97086eed307f
Oct 31 01:45:24.242381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.242342    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg" event={"ID":"7e336afe-c771-4703-8519-63dd8c922a24","Type":"ContainerStarted","Data":"487bc630ec07d1abdc88d9c0c4a466832aaa4aa07d1f059a05db18fb87740b8e"}
Oct 31 01:45:24.242855 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.242408    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg" event={"ID":"7e336afe-c771-4703-8519-63dd8c922a24","Type":"ContainerStarted","Data":"c2d649463aa54c44564f301fcd6bf3e195b6724d374c4a2b8c0c97086eed307f"}
Oct 31 01:45:24.262688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.262580    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg" podStartSLOduration=1.262567555 podStartE2EDuration="1.262567555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:24.259482532 +0000 UTC m=+1482.208130925" watchObservedRunningTime="2026-10-31 01:45:24.262567555 +0000 UTC m=+1482.211215946"
Oct 31 01:45:24.430247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.430223    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:24.542111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.541693    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-xks-ca\") pod \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") "
Oct 31 01:45:24.542111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.541896    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-kube-api-access-4pbh8\" (UniqueName: \"kubernetes.io/projected/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-kube-api-access-4pbh8\") pod \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") "
Oct 31 01:45:24.542111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.542029    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-openshift-ca\") pod \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\" (UID: \"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85\") "
Oct 31 01:45:24.542989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.542955    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-openshift-ca" pod "f3990a2d-4452-4f9b-9b13-d19e6d9d8e85" (UID: "f3990a2d-4452-4f9b-9b13-d19e6d9d8e85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:24.546394 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.546368    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-kube-api-access-4pbh8" pod "f3990a2d-4452-4f9b-9b13-d19e6d9d8e85" (UID: "f3990a2d-4452-4f9b-9b13-d19e6d9d8e85"). InnerVolumeSpecName "kube-api-access-4pbh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:24.546394 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.546380    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-xks-ca" pod "f3990a2d-4452-4f9b-9b13-d19e6d9d8e85" (UID: "f3990a2d-4452-4f9b-9b13-d19e6d9d8e85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:45:24.644299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.644268    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:24.644299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.644296    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pbh8\" (UniqueName: \"kubernetes.io/projected/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-kube-api-access-4pbh8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:24.644299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:24.644305    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:25.260055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:25.260019    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"
Oct 31 01:45:25.260488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:25.260023    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx" event={"ID":"f3990a2d-4452-4f9b-9b13-d19e6d9d8e85","Type":"ContainerDied","Data":"5bac80441ea76b42630d8e27f37767132f21b91f3ce4014c963c1b074321db46"}
Oct 31 01:45:25.260488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:25.260203    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bac80441ea76b42630d8e27f37767132f21b91f3ce4014c963c1b074321db46"
Oct 31 01:45:25.261781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:25.261756    5293 generic.go:356] "Generic (PLEG): container finished" podID="7e336afe-c771-4703-8519-63dd8c922a24" containerID="487bc630ec07d1abdc88d9c0c4a466832aaa4aa07d1f059a05db18fb87740b8e" exitCode=0
Oct 31 01:45:25.261913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:25.261784    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg" event={"ID":"7e336afe-c771-4703-8519-63dd8c922a24","Type":"ContainerDied","Data":"487bc630ec07d1abdc88d9c0c4a466832aaa4aa07d1f059a05db18fb87740b8e"}
Oct 31 01:45:26.447275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.447248    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:26.603528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.603439    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e336afe-c771-4703-8519-63dd8c922a24-kube-api-access-ldwxw\" (UniqueName: \"kubernetes.io/projected/7e336afe-c771-4703-8519-63dd8c922a24-kube-api-access-ldwxw\") pod \"7e336afe-c771-4703-8519-63dd8c922a24\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") "
Oct 31 01:45:26.603528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.603488    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-xks-ca\") pod \"7e336afe-c771-4703-8519-63dd8c922a24\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") "
Oct 31 01:45:26.603753 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.603621    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-openshift-ca\") pod \"7e336afe-c771-4703-8519-63dd8c922a24\" (UID: \"7e336afe-c771-4703-8519-63dd8c922a24\") "
Oct 31 01:45:26.603983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.603957    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-openshift-ca" pod "7e336afe-c771-4703-8519-63dd8c922a24" (UID: "7e336afe-c771-4703-8519-63dd8c922a24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:26.604509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.604487    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:26.605670 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.605648    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e336afe-c771-4703-8519-63dd8c922a24-kube-api-access-ldwxw" pod "7e336afe-c771-4703-8519-63dd8c922a24" (UID: "7e336afe-c771-4703-8519-63dd8c922a24"). InnerVolumeSpecName "kube-api-access-ldwxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:26.606061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.606021    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-xks-ca" pod "7e336afe-c771-4703-8519-63dd8c922a24" (UID: "7e336afe-c771-4703-8519-63dd8c922a24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:45:26.707521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.707475    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldwxw\" (UniqueName: \"kubernetes.io/projected/7e336afe-c771-4703-8519-63dd8c922a24-kube-api-access-ldwxw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:26.707814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:26.707787    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e336afe-c771-4703-8519-63dd8c922a24-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:27.277518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:27.277493    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"
Oct 31 01:45:27.277518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:27.277510    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg" event={"ID":"7e336afe-c771-4703-8519-63dd8c922a24","Type":"ContainerDied","Data":"c2d649463aa54c44564f301fcd6bf3e195b6724d374c4a2b8c0c97086eed307f"}
Oct 31 01:45:27.277733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:27.277552    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2d649463aa54c44564f301fcd6bf3e195b6724d374c4a2b8c0c97086eed307f"
Oct 31 01:45:30.592618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:30.592584    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:45:32.781071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:32.780368    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:32.876062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:32.875142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.110119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.108122    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.407304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.407155    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.421614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.421563    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:45:33.568198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.568140    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.569648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.569613    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.570165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.570142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.639065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.635170    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.639065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.636425    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:45:33.786278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.786146    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.848253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.848193    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.850778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.850745    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.875456 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.875397    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:45:33.876334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:33.876314    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:45:34.266066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.178831    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.302812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.302763    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:45:34.440079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.435276    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.440079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.435334    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.569120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.568602    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.580180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.572647    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:45:34.580180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.572800    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:45:34.580180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.572856    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:45:34.585178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.585136    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.585417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.585395    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.600060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.596329    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:45:34.600060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.596394    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:45:34.654072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.640511    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.725253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.640596    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.725253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.715616    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:45:34.725253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.715676    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:45:34.725253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.682982    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.854540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.853319    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.869103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.855319    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:45:34.869103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.866117    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.896067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.870566    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:45:34.896296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:34.896262    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.192338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.192294    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:45:35.272238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.272128    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 01:45:35.475113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.450175    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.475113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.452517    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:45:35.577192 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.577146    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.624066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.618749    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:45:35.624066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.619151    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:45:35.780327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.780227    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.782212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.782174    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.921185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.921142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:45:35.957658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:35.929186    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.026750 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.001332    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.320099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.292099    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:45:36.485104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.410810    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:36.485104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.410886    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.485104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.454070    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.485104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.459891    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:45:36.598163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.589828    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.619628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.619583    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.634376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.634331    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.800065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.796151    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:45:36.850121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.846167    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.917080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.917027    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"]
Oct 31 01:45:36.926065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.918201    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7e336afe-c771-4703-8519-63dd8c922a24" containerName="revoke-keys"
Oct 31 01:45:36.926602 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.926581    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e336afe-c771-4703-8519-63dd8c922a24" containerName="revoke-keys"
Oct 31 01:45:36.926728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.926717    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f3990a2d-4452-4f9b-9b13-d19e6d9d8e85" containerName="revoke-keys"
Oct 31 01:45:36.926799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.926789    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3990a2d-4452-4f9b-9b13-d19e6d9d8e85" containerName="revoke-keys"
Oct 31 01:45:36.927076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.927032    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e336afe-c771-4703-8519-63dd8c922a24" containerName="revoke-keys"
Oct 31 01:45:36.931110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.931087    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3990a2d-4452-4f9b-9b13-d19e6d9d8e85" containerName="revoke-keys"
Oct 31 01:45:36.958914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.958170    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:45:36.959829 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.959802    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"]
Oct 31 01:45:36.960170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.960150    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:36.973450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:36.973422    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9cc66e-openshift-default-dockercfg-9r6q4\""
Oct 31 01:45:37.026624 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.026578    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.094275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.094236    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:37.094604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.094572    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.110528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.110424    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.110802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.110782    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.110936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.110923    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.111172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.111144    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.111336 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.111322    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.111434 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.111421    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.111539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.111526    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.111648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.111635    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.111774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.111762    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vdh\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-kube-api-access-q4vdh\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.147177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.147131    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.213108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.213067    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.213414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.213386    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.241232 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.241190    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.242137 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.242110    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.242410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.242391    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.242523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.242510    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.242624 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.242612    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.242757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.242743    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.242885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.242871    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vdh\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-kube-api-access-q4vdh\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.243375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.213924    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.244213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.244194    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.246216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.246190    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.246631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.246582    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.247032 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.247010    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.267064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.251295    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.267064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.261596    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.267467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.267363    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.272888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.272827    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vdh\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-kube-api-access-q4vdh\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.293721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.292694    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.334694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.334211    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:37.334694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.334280    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.335487 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.335237    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:37.495067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.484145    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.495067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.485281    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.530483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.530140    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:37.530483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.530204    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.604282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.602071    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.636118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.635133    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.636288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.636115    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.814908 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.814865    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.850175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.850129    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.967222 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:37.967176    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.030252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.030193    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.035343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.035116    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.154717 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.153749    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.294240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.294140    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.487212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.487160    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.487584 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.487466    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.742075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.692095    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:45:38.742075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.693431    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.742075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.708651    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:45:38.742075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.730547    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:45:38.820962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.819532    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.932118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.886652    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:45:38.932118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:38.923879    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:45:39.111939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.111901    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:45:39.143085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.117446    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.184760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.173909    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.457855 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.457753    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:45:39.458441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.458413    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:45:39.732548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.714181    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.732548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.714477    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:45:39.795379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.795182    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:45:39.814057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.810401    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6dz48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:39.814316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.814287    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6dz48" podUID="774c137c-c103-4d59-b525-37ad6df863e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.822604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.822403    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.946718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.945262    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.993155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.986470    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:45:39.993155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:39.987909    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:40.033640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.033587    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:40.087998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.087940    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:40.088347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.088308    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:40.091500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.091478    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"]
Oct 31 01:45:40.162913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.162867    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 31 01:45:40.165595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.165563    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" event={"ID":"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff","Type":"ContainerStarted","Data":"13be72c92facfcc1bb9c68b766d08385e9a77752d161a0f2f64757083c2e8b66"}
Oct 31 01:45:40.536599 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.536540    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:40.618093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.616847    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:40.651061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.647218    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:40.664218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.664176    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:40.750196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.731384    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:40.862560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.861589    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:45:40.864182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.864148    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:45:40.864311 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.864195    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:45:40.864311 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.864228    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:45:40.875295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.875270    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:45:40.876322 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.876305    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:45:40.877673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:40.877657    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:45:41.288761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:41.288249    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" event={"ID":"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff","Type":"ContainerStarted","Data":"a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd"}
Oct 31 01:45:41.317402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:41.317348    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podStartSLOduration=6.317331323 podStartE2EDuration="6.317331323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:41.311802132 +0000 UTC m=+1499.260450525" watchObservedRunningTime="2026-10-31 01:45:41.317331323 +0000 UTC m=+1499.265979715"
Oct 31 01:45:41.336611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:41.336571    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:41.338070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:41.338030    5293 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:45:41.338228 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:41.338100    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:45:41.789603 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:45:41.789569    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:45:42.340687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:42.340658    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:43.304708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:43.304596    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:43.314012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:43.305653    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:45:44.802317 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.802285    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"]
Oct 31 01:45:44.806598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.806575    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:44.814288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.814265    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1b634e-dockercfg-5lnxm\""
Oct 31 01:45:44.817372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.817351    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1b634e\""
Oct 31 01:45:44.824674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.824651    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"]
Oct 31 01:45:44.835859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.835833    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78ms\" (UniqueName: \"kubernetes.io/projected/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-kube-api-access-c78ms\") pod \"payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw\" (UID: \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:44.835985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.835882    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw\" (UID: \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:44.861581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.861548    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"]
Oct 31 01:45:44.867144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.867119    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:44.948076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.937524    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78ms\" (UniqueName: \"kubernetes.io/projected/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-kube-api-access-c78ms\") pod \"payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw\" (UID: \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:44.948399 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.948373    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw\" (UID: \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:44.949694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.949572    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw\" (UID: \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:44.949970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.949950    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2b24bd-88c9-4d96-be81-4993e0d424a0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b634e-5754895485-wswdk\" (UID: \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:44.950542 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:44.950519    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxvw\" (UniqueName: \"kubernetes.io/projected/4b2b24bd-88c9-4d96-be81-4993e0d424a0-kube-api-access-zxxvw\") pod \"payload-processing-e2e-api-a-1b634e-5754895485-wswdk\" (UID: \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:45.064141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.054188    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxvw\" (UniqueName: \"kubernetes.io/projected/4b2b24bd-88c9-4d96-be81-4993e0d424a0-kube-api-access-zxxvw\") pod \"payload-processing-e2e-api-a-1b634e-5754895485-wswdk\" (UID: \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:45.064141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.054342    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2b24bd-88c9-4d96-be81-4993e0d424a0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b634e-5754895485-wswdk\" (UID: \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:45.081179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.075633    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2b24bd-88c9-4d96-be81-4993e0d424a0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b634e-5754895485-wswdk\" (UID: \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:45.123755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.110769    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78ms\" (UniqueName: \"kubernetes.io/projected/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-kube-api-access-c78ms\") pod \"payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw\" (UID: \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:45.123755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.120653    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:45.183955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.180570    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxvw\" (UniqueName: \"kubernetes.io/projected/4b2b24bd-88c9-4d96-be81-4993e0d424a0-kube-api-access-zxxvw\") pod \"payload-processing-e2e-api-a-1b634e-5754895485-wswdk\" (UID: \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:45.192324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.184707    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:45.192324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.186995    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"]
Oct 31 01:45:45.491220 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:45:45.491186    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51dc2ef_e480_4cbc_8b4a_9b3ebbcc75ce.slice/crio-447dc90e90b235faf5581af8c73c04954c1b4ed439ab9a38d35981131f3a5f63 WatchSource:0}: Error finding container 447dc90e90b235faf5581af8c73c04954c1b4ed439ab9a38d35981131f3a5f63: Status 404 returned error can't find the container with id 447dc90e90b235faf5581af8c73c04954c1b4ed439ab9a38d35981131f3a5f63
Oct 31 01:45:45.491330 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.491304    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"]
Oct 31 01:45:45.493345 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:45:45.493321    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2b24bd_88c9_4d96_be81_4993e0d424a0.slice/crio-0811ae085d6d939af86c1bf4f6dbc856c770765b4536eec4a8d0b2676d8a9e7c WatchSource:0}: Error finding container 0811ae085d6d939af86c1bf4f6dbc856c770765b4536eec4a8d0b2676d8a9e7c: Status 404 returned error can't find the container with id 0811ae085d6d939af86c1bf4f6dbc856c770765b4536eec4a8d0b2676d8a9e7c
Oct 31 01:45:45.496218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.496192    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"]
Oct 31 01:45:45.828424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.828348    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"]
Oct 31 01:45:45.835321 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.835295    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:45.842954 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.842929    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9cc66e\""
Oct 31 01:45:45.856591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.856565    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"]
Oct 31 01:45:45.896266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.896236    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fntz\" (UniqueName: \"kubernetes.io/projected/d2df0794-543e-4701-bddb-e7b14ae8f8d7-kube-api-access-5fntz\") pod \"payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr\" (UID: \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:45.896447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:45.896419    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df0794-543e-4701-bddb-e7b14ae8f8d7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr\" (UID: \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:46.019677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.004267    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fntz\" (UniqueName: \"kubernetes.io/projected/d2df0794-543e-4701-bddb-e7b14ae8f8d7-kube-api-access-5fntz\") pod \"payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr\" (UID: \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:46.020334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.020269    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df0794-543e-4701-bddb-e7b14ae8f8d7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr\" (UID: \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:46.022966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.022941    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df0794-543e-4701-bddb-e7b14ae8f8d7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr\" (UID: \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:46.068065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.045619    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fntz\" (UniqueName: \"kubernetes.io/projected/d2df0794-543e-4701-bddb-e7b14ae8f8d7-kube-api-access-5fntz\") pod \"payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr\" (UID: \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\") " pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:46.162759 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.162714    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:46.335556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.334768    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk" event={"ID":"4b2b24bd-88c9-4d96-be81-4993e0d424a0","Type":"ContainerStarted","Data":"17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679"}
Oct 31 01:45:46.335556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.334835    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk" event={"ID":"4b2b24bd-88c9-4d96-be81-4993e0d424a0","Type":"ContainerStarted","Data":"0811ae085d6d939af86c1bf4f6dbc856c770765b4536eec4a8d0b2676d8a9e7c"}
Oct 31 01:45:46.335556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.335077    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:46.337967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.337566    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw" event={"ID":"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce","Type":"ContainerStarted","Data":"0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a"}
Oct 31 01:45:46.337967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.337615    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:46.337967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.337635    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw" event={"ID":"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce","Type":"ContainerStarted","Data":"447dc90e90b235faf5581af8c73c04954c1b4ed439ab9a38d35981131f3a5f63"}
Oct 31 01:45:46.363919 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.362794    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk" podStartSLOduration=2.362774336 podStartE2EDuration="2.362774336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:46.3535859 +0000 UTC m=+1504.302234293" watchObservedRunningTime="2026-10-31 01:45:46.362774336 +0000 UTC m=+1504.311422728"
Oct 31 01:45:46.377013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.376957    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw" podStartSLOduration=2.376941139 podStartE2EDuration="2.376941139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:46.372168814 +0000 UTC m=+1504.320817205" watchObservedRunningTime="2026-10-31 01:45:46.376941139 +0000 UTC m=+1504.325589534"
Oct 31 01:45:46.592228 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:46.592195    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"]
Oct 31 01:45:46.593256 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:45:46.593231    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2df0794_543e_4701_bddb_e7b14ae8f8d7.slice/crio-7c3c6947e252f1d68bc6b9a8e7a6fb34d1f763d6ef24c6fe1d1499d6c5e76a21 WatchSource:0}: Error finding container 7c3c6947e252f1d68bc6b9a8e7a6fb34d1f763d6ef24c6fe1d1499d6c5e76a21: Status 404 returned error can't find the container with id 7c3c6947e252f1d68bc6b9a8e7a6fb34d1f763d6ef24c6fe1d1499d6c5e76a21
Oct 31 01:45:47.419077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:47.418983    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr" event={"ID":"d2df0794-543e-4701-bddb-e7b14ae8f8d7","Type":"ContainerStarted","Data":"c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246"}
Oct 31 01:45:47.420058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:47.419686    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr" event={"ID":"d2df0794-543e-4701-bddb-e7b14ae8f8d7","Type":"ContainerStarted","Data":"7c3c6947e252f1d68bc6b9a8e7a6fb34d1f763d6ef24c6fe1d1499d6c5e76a21"}
Oct 31 01:45:47.420703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:47.420601    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:47.451708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:47.451611    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr" podStartSLOduration=2.451588845 podStartE2EDuration="2.451588845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:47.44109384 +0000 UTC m=+1505.389742233" watchObservedRunningTime="2026-10-31 01:45:47.451588845 +0000 UTC m=+1505.400237237"
Oct 31 01:45:55.170398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.170351    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"]
Oct 31 01:45:55.177301 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.177270    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.215140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.207865    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"]
Oct 31 01:45:55.267493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.267453    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.267671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.267534    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.267671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.267588    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ade77b0-64a0-447b-ba6a-a8199bb33298-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.267756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.267681    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.267756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.267729    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.267881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.267853    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.268030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.268009    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.268120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.268058    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsr29\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-kube-api-access-wsr29\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.268199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.268179    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.375068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.374949    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.375263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.375199    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.375263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.375250    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.375379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.375292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ade77b0-64a0-447b-ba6a-a8199bb33298-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.375379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.375341    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.375491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.375387    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.375491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.375441    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.376067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.375594    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.376067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.375637    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsr29\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-kube-api-access-wsr29\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.389063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.381746    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.389063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.383397    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.389063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.383928    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.396073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.395912    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.397075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.396992    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ade77b0-64a0-447b-ba6a-a8199bb33298-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.400120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.400026    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.417063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.403957    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsr29\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-kube-api-access-wsr29\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.417063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.414698    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.426841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.421504    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.525683 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.525316    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:55.944119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.924265    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"]
Oct 31 01:45:55.966994 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:45:55.966963    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ade77b0_64a0_447b_ba6a_a8199bb33298.slice/crio-1db8729efd0aab2cd0b50cc6ca502de633131be42bebc563bc624c34416c59e5 WatchSource:0}: Error finding container 1db8729efd0aab2cd0b50cc6ca502de633131be42bebc563bc624c34416c59e5: Status 404 returned error can't find the container with id 1db8729efd0aab2cd0b50cc6ca502de633131be42bebc563bc624c34416c59e5
Oct 31 01:45:55.973685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.973138    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:55.973685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.973229    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:55.973685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:55.973274    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:45:56.559627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:56.559592    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" event={"ID":"9ade77b0-64a0-447b-ba6a-a8199bb33298","Type":"ContainerStarted","Data":"2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132"}
Oct 31 01:45:56.560136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:56.559642    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" event={"ID":"9ade77b0-64a0-447b-ba6a-a8199bb33298","Type":"ContainerStarted","Data":"1db8729efd0aab2cd0b50cc6ca502de633131be42bebc563bc624c34416c59e5"}
Oct 31 01:45:56.582690 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:56.582641    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podStartSLOduration=1.582623329 podStartE2EDuration="1.582623329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:56.580676692 +0000 UTC m=+1514.529325095" watchObservedRunningTime="2026-10-31 01:45:56.582623329 +0000 UTC m=+1514.531271722"
Oct 31 01:45:57.439223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:57.439191    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:45:57.439409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:57.439320    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:45:57.529239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:57.529204    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:45:57.530688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:57.530657    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:45:57.530786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:57.530712    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:45:58.530063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:58.529456    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:45:58.530063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:58.529515    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:45:59.481528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:59.481382    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:45:59.536712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:59.536674    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:45:59.537342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:59.537309    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:45:59.848808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:59.848718    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"]
Oct 31 01:45:59.859694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:59.859650    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"]
Oct 31 01:45:59.859954 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:59.859880    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:45:59.865743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:45:59.865690    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b6b833-openshift-default-dockercfg-x4gvn\""
Oct 31 01:46:00.049092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.021425    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-socket\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.074121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.074086    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istiod-ca-cert\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.074478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.074435    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-certs\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.074683 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.074664    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-credential-socket\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.082193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.082159    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-token\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.120285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.120184    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-envoy\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.120557 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.120518    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-data\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.120765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.120733    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dzwl\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-kube-api-access-4dzwl\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.120955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.120940    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-podinfo\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.248809    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-envoy\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.248865    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-data\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.248955    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzwl\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-kube-api-access-4dzwl\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.249024    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-podinfo\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.249118    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-socket\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.249305    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istiod-ca-cert\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.249383    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-certs\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.249423    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-credential-socket\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.249974    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-token\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.259063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.254071    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-data\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.266086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.262578    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-socket\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.266086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.262830    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-certs\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.266086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.263757    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istiod-ca-cert\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.291916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.286241    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-credential-socket\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.291916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.288394    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-envoy\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.291916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.288954    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-token\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.291916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.291334    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzwl\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-kube-api-access-4dzwl\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.293729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.293706    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-podinfo\") pod \"e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") " pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.534907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.534447    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:46:00.534907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.534521    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:46:00.564064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.559539    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:00.895514 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:00.895479    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d50b4a_007a_4c9e_9bc6_3f9abe048781.slice/crio-63fa1013ab3bf6415266daa9e06070e83fa2a9dfd90aa57990a102f97da1ee15 WatchSource:0}: Error finding container 63fa1013ab3bf6415266daa9e06070e83fa2a9dfd90aa57990a102f97da1ee15: Status 404 returned error can't find the container with id 63fa1013ab3bf6415266daa9e06070e83fa2a9dfd90aa57990a102f97da1ee15
Oct 31 01:46:00.895856 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.895805    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"]
Oct 31 01:46:00.928846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.928799    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:00.928998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.928892    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:00.928998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:00.928938    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:01.526947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:01.526429    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:46:01.526947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:01.526490    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:46:01.641372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:01.641333    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" event={"ID":"12d50b4a-007a-4c9e-9bc6-3f9abe048781","Type":"ContainerStarted","Data":"902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1"}
Oct 31 01:46:01.641758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:01.641384    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" event={"ID":"12d50b4a-007a-4c9e-9bc6-3f9abe048781","Type":"ContainerStarted","Data":"63fa1013ab3bf6415266daa9e06070e83fa2a9dfd90aa57990a102f97da1ee15"}
Oct 31 01:46:01.667657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:01.667616    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podStartSLOduration=2.667602711 podStartE2EDuration="2.667602711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:01.663423963 +0000 UTC m=+1519.612072354" watchObservedRunningTime="2026-10-31 01:46:01.667602711 +0000 UTC m=+1519.616251103"
Oct 31 01:46:02.526414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:02.526379    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:46:02.526575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:02.526434    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:46:02.560188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:02.560161    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:02.565504 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:02.565482    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:02.652025 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:02.651984    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:02.652936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:02.652917    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:03.531208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:03.531175    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:46:03.531500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:03.531472    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:46:04.558089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:04.558053    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:46:04.558688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:04.558665    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:46:04.559790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:04.559770    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:46:05.948214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:05.948180    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"]
Oct 31 01:46:05.953447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:05.953426    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:05.962101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:05.962073    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b6b833-dockercfg-zrjmt\""
Oct 31 01:46:05.962351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:05.962329    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b6b833\""
Oct 31 01:46:05.963116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:05.963094    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"]
Oct 31 01:46:05.975673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:05.975652    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"]
Oct 31 01:46:05.981694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:05.981674    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:05.990517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:05.990494    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"]
Oct 31 01:46:06.068243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.068138    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b801913c-b11c-49f6-9e07-7cf83fb28d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st\" (UID: \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:06.068612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.068586    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjrg\" (UniqueName: \"kubernetes.io/projected/07c887a3-622c-4808-b16a-c0b5d359f3fd-kube-api-access-7pjrg\") pod \"payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9\" (UID: \"07c887a3-622c-4808-b16a-c0b5d359f3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:06.079366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.079333    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6h4\" (UniqueName: \"kubernetes.io/projected/b801913c-b11c-49f6-9e07-7cf83fb28d7f-kube-api-access-bj6h4\") pod \"payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st\" (UID: \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:06.083140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.079810    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c887a3-622c-4808-b16a-c0b5d359f3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9\" (UID: \"07c887a3-622c-4808-b16a-c0b5d359f3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:06.234253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.234135    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b801913c-b11c-49f6-9e07-7cf83fb28d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st\" (UID: \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:06.234600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.234570    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjrg\" (UniqueName: \"kubernetes.io/projected/07c887a3-622c-4808-b16a-c0b5d359f3fd-kube-api-access-7pjrg\") pod \"payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9\" (UID: \"07c887a3-622c-4808-b16a-c0b5d359f3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:06.234769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.234754    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6h4\" (UniqueName: \"kubernetes.io/projected/b801913c-b11c-49f6-9e07-7cf83fb28d7f-kube-api-access-bj6h4\") pod \"payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st\" (UID: \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:06.242070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.235231    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c887a3-622c-4808-b16a-c0b5d359f3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9\" (UID: \"07c887a3-622c-4808-b16a-c0b5d359f3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:06.244729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.243596    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b801913c-b11c-49f6-9e07-7cf83fb28d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st\" (UID: \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:06.245757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.245732    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c887a3-622c-4808-b16a-c0b5d359f3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9\" (UID: \"07c887a3-622c-4808-b16a-c0b5d359f3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:06.295082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.285177    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjrg\" (UniqueName: \"kubernetes.io/projected/07c887a3-622c-4808-b16a-c0b5d359f3fd-kube-api-access-7pjrg\") pod \"payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9\" (UID: \"07c887a3-622c-4808-b16a-c0b5d359f3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:06.295082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.289736    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6h4\" (UniqueName: \"kubernetes.io/projected/b801913c-b11c-49f6-9e07-7cf83fb28d7f-kube-api-access-bj6h4\") pod \"payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st\" (UID: \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:06.309618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.309330    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:06.582261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.582199    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:06.763851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.763820    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"]
Oct 31 01:46:06.765320 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:06.765296    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb801913c_b11c_49f6_9e07_7cf83fb28d7f.slice/crio-7feec3849795c09ab6bd2b6704850e0bf26659fd2cc144d82a663f89a12faa5c WatchSource:0}: Error finding container 7feec3849795c09ab6bd2b6704850e0bf26659fd2cc144d82a663f89a12faa5c: Status 404 returned error can't find the container with id 7feec3849795c09ab6bd2b6704850e0bf26659fd2cc144d82a663f89a12faa5c
Oct 31 01:46:06.902983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:06.902949    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"]
Oct 31 01:46:06.907709 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:06.907684    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c887a3_622c_4808_b16a_c0b5d359f3fd.slice/crio-1b37eff617b390a589df78191a5acc9f2f79ff360b9ce0d6d35c730dc2ae65cd WatchSource:0}: Error finding container 1b37eff617b390a589df78191a5acc9f2f79ff360b9ce0d6d35c730dc2ae65cd: Status 404 returned error can't find the container with id 1b37eff617b390a589df78191a5acc9f2f79ff360b9ce0d6d35c730dc2ae65cd
Oct 31 01:46:07.697860 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:07.697814    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st" event={"ID":"b801913c-b11c-49f6-9e07-7cf83fb28d7f","Type":"ContainerStarted","Data":"66b1e5e165ff79f9c7d5280c352a7d0d241a1c299b943e97fee0455e9c0f027f"}
Oct 31 01:46:07.698429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:07.697898    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st" event={"ID":"b801913c-b11c-49f6-9e07-7cf83fb28d7f","Type":"ContainerStarted","Data":"7feec3849795c09ab6bd2b6704850e0bf26659fd2cc144d82a663f89a12faa5c"}
Oct 31 01:46:07.698429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:07.698190    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:07.700088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:07.700064    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9" event={"ID":"07c887a3-622c-4808-b16a-c0b5d359f3fd","Type":"ContainerStarted","Data":"6049b1d4668787da89d71588337e89cd2a1422588007b8a5d63a414503410351"}
Oct 31 01:46:07.700215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:07.700106    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9" event={"ID":"07c887a3-622c-4808-b16a-c0b5d359f3fd","Type":"ContainerStarted","Data":"1b37eff617b390a589df78191a5acc9f2f79ff360b9ce0d6d35c730dc2ae65cd"}
Oct 31 01:46:07.700215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:07.700130    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:07.719805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:07.719757    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st" podStartSLOduration=2.719744547 podStartE2EDuration="2.719744547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:07.716222827 +0000 UTC m=+1525.664871219" watchObservedRunningTime="2026-10-31 01:46:07.719744547 +0000 UTC m=+1525.668392955"
Oct 31 01:46:07.737766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:07.737722    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9" podStartSLOduration=2.737706567 podStartE2EDuration="2.737706567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:07.734307327 +0000 UTC m=+1525.682955718" watchObservedRunningTime="2026-10-31 01:46:07.737706567 +0000 UTC m=+1525.686354958"
Oct 31 01:46:09.830064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:09.829169    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"]
Oct 31 01:46:09.866737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:09.866709    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"]
Oct 31 01:46:09.866925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:09.866809    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:09.871320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:09.871295    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0239db-openshift-default-dockercfg-9xc85\""
Oct 31 01:46:10.101116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.086870    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-certs\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.101936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.101897    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-token\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.102242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.102221    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istiod-ca-cert\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.102527 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.102411    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-data\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.102799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.102757    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpkh\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-kube-api-access-qrpkh\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.103158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.103112    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-envoy\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.103279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.103207    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-podinfo\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.103279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.103249    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-socket\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.103279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.103275    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-credential-socket\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.235326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.235289    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-envoy\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.236172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.236148    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-podinfo\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.236959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.236938    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-socket\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.237597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.237577    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-credential-socket\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.238000 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.237979    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-certs\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.238202 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.238176    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-token\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.238321 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.238304    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istiod-ca-cert\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.238437 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.238424    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-data\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.238612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.238599    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpkh\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-kube-api-access-qrpkh\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.251092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.241191    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-certs\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.251092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.237523    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-socket\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.251092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.241464    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-credential-socket\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.251092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.242350    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istiod-ca-cert\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.251092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.243106    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-data\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.263136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.259943    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpkh\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-kube-api-access-qrpkh\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.268105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.268032    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-token\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.276792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.276765    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-envoy\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.279032 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.278990    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-podinfo\") pod \"e2e-api-b-0239db-openshift-default-5cb4966c67-nq994\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") " pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.486425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.486393    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:10.600106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.600073    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:46:10.932639 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:10.932595    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb89a8ad_f1ab_4f91_a4e2_afd6e24b6fc0.slice/crio-0123f8cca44516d7e2b0d368d63b0aa6d66530d8a42f6096aaccfa0b5b490ecc WatchSource:0}: Error finding container 0123f8cca44516d7e2b0d368d63b0aa6d66530d8a42f6096aaccfa0b5b490ecc: Status 404 returned error can't find the container with id 0123f8cca44516d7e2b0d368d63b0aa6d66530d8a42f6096aaccfa0b5b490ecc
Oct 31 01:46:10.944068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.941990    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"]
Oct 31 01:46:10.961798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.956481    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:10.961798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.956571    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:10.961798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:10.956616    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:11.829932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:11.829896    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" event={"ID":"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0","Type":"ContainerStarted","Data":"fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387"}
Oct 31 01:46:11.830442 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:11.830417    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" event={"ID":"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0","Type":"ContainerStarted","Data":"0123f8cca44516d7e2b0d368d63b0aa6d66530d8a42f6096aaccfa0b5b490ecc"}
Oct 31 01:46:11.855347 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:46:11.855263    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:46:11.862030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:11.861972    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" podStartSLOduration=2.861952912 podStartE2EDuration="2.861952912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:11.855688121 +0000 UTC m=+1529.804336524" watchObservedRunningTime="2026-10-31 01:46:11.861952912 +0000 UTC m=+1529.810601305"
Oct 31 01:46:12.487529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:12.487495    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:12.493264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:12.493239    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:12.847963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:12.847878    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:12.848860 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:12.848840    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:46:15.733327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.733296    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"]
Oct 31 01:46:15.740010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.739975    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:15.746951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.746897    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0239db\""
Oct 31 01:46:15.752445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.750159    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"]
Oct 31 01:46:15.762802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.761805    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nkl\" (UniqueName: \"kubernetes.io/projected/2a453023-ea17-4aa3-917e-f404ba3bfabe-kube-api-access-w9nkl\") pod \"payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n\" (UID: \"2a453023-ea17-4aa3-917e-f404ba3bfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:15.762802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.762174    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a453023-ea17-4aa3-917e-f404ba3bfabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n\" (UID: \"2a453023-ea17-4aa3-917e-f404ba3bfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:15.774604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.774581    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"]
Oct 31 01:46:15.779299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.779278    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:15.783338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.783319    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0239db-dockercfg-lmlfl\""
Oct 31 01:46:15.793338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.791326    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"]
Oct 31 01:46:15.942089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.905973    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a453023-ea17-4aa3-917e-f404ba3bfabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n\" (UID: \"2a453023-ea17-4aa3-917e-f404ba3bfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:15.942089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.906070    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47v7\" (UniqueName: \"kubernetes.io/projected/8e81f600-b675-4502-b60b-44b3d82e8b27-kube-api-access-p47v7\") pod \"payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz\" (UID: \"8e81f600-b675-4502-b60b-44b3d82e8b27\") " pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:15.942089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.906246    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81f600-b675-4502-b60b-44b3d82e8b27-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz\" (UID: \"8e81f600-b675-4502-b60b-44b3d82e8b27\") " pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:15.942089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.906402    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nkl\" (UniqueName: \"kubernetes.io/projected/2a453023-ea17-4aa3-917e-f404ba3bfabe-kube-api-access-w9nkl\") pod \"payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n\" (UID: \"2a453023-ea17-4aa3-917e-f404ba3bfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:15.968078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:15.968023    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a453023-ea17-4aa3-917e-f404ba3bfabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n\" (UID: \"2a453023-ea17-4aa3-917e-f404ba3bfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:16.163377 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.163340    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81f600-b675-4502-b60b-44b3d82e8b27-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz\" (UID: \"8e81f600-b675-4502-b60b-44b3d82e8b27\") " pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:16.165526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.165448    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47v7\" (UniqueName: \"kubernetes.io/projected/8e81f600-b675-4502-b60b-44b3d82e8b27-kube-api-access-p47v7\") pod \"payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz\" (UID: \"8e81f600-b675-4502-b60b-44b3d82e8b27\") " pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:16.179463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.164528    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81f600-b675-4502-b60b-44b3d82e8b27-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz\" (UID: \"8e81f600-b675-4502-b60b-44b3d82e8b27\") " pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:16.228619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.228588    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47v7\" (UniqueName: \"kubernetes.io/projected/8e81f600-b675-4502-b60b-44b3d82e8b27-kube-api-access-p47v7\") pod \"payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz\" (UID: \"8e81f600-b675-4502-b60b-44b3d82e8b27\") " pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:16.254587 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.254558    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nkl\" (UniqueName: \"kubernetes.io/projected/2a453023-ea17-4aa3-917e-f404ba3bfabe-kube-api-access-w9nkl\") pod \"payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n\" (UID: \"2a453023-ea17-4aa3-917e-f404ba3bfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:16.366753 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.366724    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:16.427109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.426326    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:16.848744 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:16.843225    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a453023_ea17_4aa3_917e_f404ba3bfabe.slice/crio-1ce2ebd34553467f816ea03647a2da3b9a14596a4c06880df3a0d9561888e8aa WatchSource:0}: Error finding container 1ce2ebd34553467f816ea03647a2da3b9a14596a4c06880df3a0d9561888e8aa: Status 404 returned error can't find the container with id 1ce2ebd34553467f816ea03647a2da3b9a14596a4c06880df3a0d9561888e8aa
Oct 31 01:46:16.880105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.863404    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"]
Oct 31 01:46:16.951350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:16.951250    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"]
Oct 31 01:46:17.170969 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:17.170935    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n" event={"ID":"2a453023-ea17-4aa3-917e-f404ba3bfabe","Type":"ContainerStarted","Data":"a1b9a8f869f9462e0a91b2b3463b1d3da08c27a2e01ea5451185954d31dbb99a"}
Oct 31 01:46:17.171188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:17.170999    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:17.171188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:17.171026    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n" event={"ID":"2a453023-ea17-4aa3-917e-f404ba3bfabe","Type":"ContainerStarted","Data":"1ce2ebd34553467f816ea03647a2da3b9a14596a4c06880df3a0d9561888e8aa"}
Oct 31 01:46:17.172672 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:17.172649    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz" event={"ID":"8e81f600-b675-4502-b60b-44b3d82e8b27","Type":"ContainerStarted","Data":"bc67b856ec426647ef166b2d808a70acb21ac1f2a8745040b9c4038e8554dbbf"}
Oct 31 01:46:17.172791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:17.172687    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz" event={"ID":"8e81f600-b675-4502-b60b-44b3d82e8b27","Type":"ContainerStarted","Data":"8f4c4fcb1187c04de0ca9a5dd4ed1c0a7d71aa2c6474983f811c73f1d1db9647"}
Oct 31 01:46:17.172791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:17.172734    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:17.191095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:17.191015    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n" podStartSLOduration=2.191000118 podStartE2EDuration="2.191000118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:17.189370877 +0000 UTC m=+1535.138019269" watchObservedRunningTime="2026-10-31 01:46:17.191000118 +0000 UTC m=+1535.139648510"
Oct 31 01:46:17.233281 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:17.233219    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz" podStartSLOduration=2.2331974199999998 podStartE2EDuration="2.23319742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:17.220443939 +0000 UTC m=+1535.169092331" watchObservedRunningTime="2026-10-31 01:46:17.23319742 +0000 UTC m=+1535.181845816"
Oct 31 01:46:18.730704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:18.730672    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:46:18.732271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:18.732248    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:46:19.085526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.078240    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"]
Oct 31 01:46:19.092727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.092701    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.093397 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.093340    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"]
Oct 31 01:46:19.098921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.098900    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.110949 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.110923    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"]
Oct 31 01:46:19.147318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.147265    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"]
Oct 31 01:46:19.189054 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.189008    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1b634e-openshift-default-dockercfg-qdmlf\""
Oct 31 01:46:19.192602 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.192572    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.192739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.192637    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.192739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.192697    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.192861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.192756    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpcg\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-kube-api-access-2cpcg\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.192861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.192821    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.192861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.192852    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.193017 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.192871    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.193017 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.192898    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.193148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193019    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.193148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193080    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.193258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193172    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.193258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193221    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.193327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193298    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.193372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193333    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.193410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193378    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7q5w\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-kube-api-access-p7q5w\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.193512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193496    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.193561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193547    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e99c30-f334-44be-b22b-a1e12acd6762-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.193654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.193643    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.295292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295260    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpcg\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-kube-api-access-2cpcg\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.295292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295298    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.295529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295327    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.295529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295356    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.295529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295392    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.295529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295427    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.295529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295460    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.295823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295791    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.295918 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295847    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.295918 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295848    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.296033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295968    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.296033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.295997    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.296165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296102    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.296165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296118    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.296339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296179    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.296339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296228    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.296339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296251    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7q5w\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-kube-api-access-p7q5w\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.296501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296327    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.296561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296526    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e99c30-f334-44be-b22b-a1e12acd6762-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.296657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296638    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.296728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296701    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.296728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296705    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.296830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296785    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.296888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.296825    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.297331 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.297305    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.297568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.297544    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.297734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.297664    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.298806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.298761    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e99c30-f334-44be-b22b-a1e12acd6762-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.299138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.299111    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.299585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.299547    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.300026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.300005    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.300726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.300705    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.307927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.307881    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpcg\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-kube-api-access-2cpcg\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.308535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.308514    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.308798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.308778    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:19.309966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.309946    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7q5w\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-kube-api-access-p7q5w\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.464017 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.463981    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:19.468210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:19.468186    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:20.022078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.013094    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"]
Oct 31 01:46:20.052953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.052904    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:20.053118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.052995    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:20.053118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.053061    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:20.066604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.066574    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"]
Oct 31 01:46:20.070273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.069702    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:20.070273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.069785    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:20.070273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.069826    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:20.211089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.205794    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" event={"ID":"5df3b6eb-f9da-446a-a551-3c56f0a3615c","Type":"ContainerStarted","Data":"07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401"}
Oct 31 01:46:20.211089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.205856    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" event={"ID":"5df3b6eb-f9da-446a-a551-3c56f0a3615c","Type":"ContainerStarted","Data":"b77abe1fbf95f4134136c49d726f57d5ac0956ab14d1e39b2bd05efc5f26bcb8"}
Oct 31 01:46:20.211089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.209055    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" event={"ID":"c7e99c30-f334-44be-b22b-a1e12acd6762","Type":"ContainerStarted","Data":"5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb"}
Oct 31 01:46:20.211089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.209104    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" event={"ID":"c7e99c30-f334-44be-b22b-a1e12acd6762","Type":"ContainerStarted","Data":"e77dbd32ac148e768ca95001a3fd4878ba061900abc9b400b4807057fe212e50"}
Oct 31 01:46:20.233971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.233918    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podStartSLOduration=1.233489996 podStartE2EDuration="1.233489996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:20.228340158 +0000 UTC m=+1538.176988550" watchObservedRunningTime="2026-10-31 01:46:20.233489996 +0000 UTC m=+1538.182138392"
Oct 31 01:46:20.259698 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.259581    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podStartSLOduration=1.259562959 podStartE2EDuration="1.259562959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:20.255204139 +0000 UTC m=+1538.203852529" watchObservedRunningTime="2026-10-31 01:46:20.259562959 +0000 UTC m=+1538.208211356"
Oct 31 01:46:20.464851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.464810    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:20.468547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:20.468520    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:21.482561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:21.482171    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:46:21.482561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:21.482235    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:46:21.496084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:21.491833    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:46:21.496084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:21.491895    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:46:22.516123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:22.493385    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:46:22.516123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:22.493447    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:46:22.516123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:22.495974    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:46:22.516123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:22.496025    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:46:23.466103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:23.464449    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:46:23.466103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:23.464518    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:46:23.472348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:23.471153    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:46:23.472348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:23.471210    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:46:24.464951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.464514    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:46:24.464951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.464577    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:46:24.468840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.468816    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:46:24.468930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.468858    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:46:24.970989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.970958    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"]
Oct 31 01:46:24.976733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.976716    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:24.980915 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.980857    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:46:24.981344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.981322    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:46:24.981583 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.981556    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:46:24.981687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.981660    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:46:24.988087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:24.988063    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"]
Oct 31 01:46:25.138066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.121114    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500852e7-8968-4874-9992-6086ef6e3755-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.138066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.121273    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwt9n\" (UniqueName: \"kubernetes.io/projected/500852e7-8968-4874-9992-6086ef6e3755-kube-api-access-jwt9n\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.138066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.121431    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500852e7-8968-4874-9992-6086ef6e3755-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.228117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.225086    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500852e7-8968-4874-9992-6086ef6e3755-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.228117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.225297    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500852e7-8968-4874-9992-6086ef6e3755-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.228117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.225429    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwt9n\" (UniqueName: \"kubernetes.io/projected/500852e7-8968-4874-9992-6086ef6e3755-kube-api-access-jwt9n\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.228117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.226741    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500852e7-8968-4874-9992-6086ef6e3755-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.251930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.251894    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500852e7-8968-4874-9992-6086ef6e3755-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.259718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.259675    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwt9n\" (UniqueName: \"kubernetes.io/projected/500852e7-8968-4874-9992-6086ef6e3755-kube-api-access-jwt9n\") pod \"maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.363657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.359137    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:25.464741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.464705    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:46:25.464935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.464766    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:46:25.470071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.469979    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:46:25.475075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.470575    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:46:25.751156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:25.751078    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"]
Oct 31 01:46:25.757241 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:25.757214    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500852e7_8968_4874_9992_6086ef6e3755.slice/crio-0e2de1ad1b4654cb1c763abf7086a0fafb4fdc5611edcd860da54f1593404c22 WatchSource:0}: Error finding container 0e2de1ad1b4654cb1c763abf7086a0fafb4fdc5611edcd860da54f1593404c22: Status 404 returned error can't find the container with id 0e2de1ad1b4654cb1c763abf7086a0fafb4fdc5611edcd860da54f1593404c22
Oct 31 01:46:26.465582 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:26.465516    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:46:26.466161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:26.466121    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:46:26.468887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:26.468860    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:46:26.469005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:26.468904    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:46:26.651062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:26.650833    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l" event={"ID":"500852e7-8968-4874-9992-6086ef6e3755","Type":"ContainerStarted","Data":"6fa4ec8dd7bf804ff131444146ebf5d43cc9212e32490fc00bb496d4f4d42ee5"}
Oct 31 01:46:26.651062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:26.650902    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l" event={"ID":"500852e7-8968-4874-9992-6086ef6e3755","Type":"ContainerStarted","Data":"0e2de1ad1b4654cb1c763abf7086a0fafb4fdc5611edcd860da54f1593404c22"}
Oct 31 01:46:26.721980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:26.721856    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l" podStartSLOduration=2.721834598 podStartE2EDuration="2.721834598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:26.706389612 +0000 UTC m=+1544.655038004" watchObservedRunningTime="2026-10-31 01:46:26.721834598 +0000 UTC m=+1544.670482990"
Oct 31 01:46:27.494675 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:27.494639    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:27.496518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:27.496330    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:27.497872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:27.497752    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:46:27.498029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:27.497962    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:27.498546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:27.498525    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:27.499435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:27.499414    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:46:27.663262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:27.663225    5293 generic.go:356] "Generic (PLEG): container finished" podID="500852e7-8968-4874-9992-6086ef6e3755" containerID="6fa4ec8dd7bf804ff131444146ebf5d43cc9212e32490fc00bb496d4f4d42ee5" exitCode=0
Oct 31 01:46:27.663688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:27.663384    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l" event={"ID":"500852e7-8968-4874-9992-6086ef6e3755","Type":"ContainerDied","Data":"6fa4ec8dd7bf804ff131444146ebf5d43cc9212e32490fc00bb496d4f4d42ee5"}
Oct 31 01:46:28.243190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.237307    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:46:28.243190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.237705    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:46:28.307748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.307400    5293 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:46:28.307748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.307460    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:46:28.674891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.674859    5293 generic.go:356] "Generic (PLEG): container finished" podID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerID="a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd" exitCode=0
Oct 31 01:46:28.675314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.674936    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" event={"ID":"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff","Type":"ContainerDied","Data":"a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd"}
Oct 31 01:46:28.675718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.675694    5293 scope.go:122] "RemoveContainer" containerID="a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd"
Oct 31 01:46:28.677635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.677602    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:28.677775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.677754    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:28.677827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:28.677811    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:29.051155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.051122    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:29.191971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.191938    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500852e7-8968-4874-9992-6086ef6e3755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500852e7-8968-4874-9992-6086ef6e3755-maas-api-xks-ca\") pod \"500852e7-8968-4874-9992-6086ef6e3755\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") "
Oct 31 01:46:29.192262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.192247    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500852e7-8968-4874-9992-6086ef6e3755-kube-api-access-jwt9n\" (UniqueName: \"kubernetes.io/projected/500852e7-8968-4874-9992-6086ef6e3755-kube-api-access-jwt9n\") pod \"500852e7-8968-4874-9992-6086ef6e3755\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") "
Oct 31 01:46:29.192339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.192319    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500852e7-8968-4874-9992-6086ef6e3755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500852e7-8968-4874-9992-6086ef6e3755-maas-api-openshift-ca\") pod \"500852e7-8968-4874-9992-6086ef6e3755\" (UID: \"500852e7-8968-4874-9992-6086ef6e3755\") "
Oct 31 01:46:29.192820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.192789    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500852e7-8968-4874-9992-6086ef6e3755-maas-api-openshift-ca" pod "500852e7-8968-4874-9992-6086ef6e3755" (UID: "500852e7-8968-4874-9992-6086ef6e3755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:46:29.194574 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.194543    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500852e7-8968-4874-9992-6086ef6e3755-maas-api-xks-ca" pod "500852e7-8968-4874-9992-6086ef6e3755" (UID: "500852e7-8968-4874-9992-6086ef6e3755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:46:29.194781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.194760    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500852e7-8968-4874-9992-6086ef6e3755-kube-api-access-jwt9n" pod "500852e7-8968-4874-9992-6086ef6e3755" (UID: "500852e7-8968-4874-9992-6086ef6e3755"). InnerVolumeSpecName "kube-api-access-jwt9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:29.295572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.295486    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwt9n\" (UniqueName: \"kubernetes.io/projected/500852e7-8968-4874-9992-6086ef6e3755-kube-api-access-jwt9n\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:29.295572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.295521    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500852e7-8968-4874-9992-6086ef6e3755-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:29.295572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.295536    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500852e7-8968-4874-9992-6086ef6e3755-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:29.785642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.785602    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l" event={"ID":"500852e7-8968-4874-9992-6086ef6e3755","Type":"ContainerDied","Data":"0e2de1ad1b4654cb1c763abf7086a0fafb4fdc5611edcd860da54f1593404c22"}
Oct 31 01:46:29.786892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.786870    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e2de1ad1b4654cb1c763abf7086a0fafb4fdc5611edcd860da54f1593404c22"
Oct 31 01:46:29.787139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.786510    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"
Oct 31 01:46:29.791468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:29.791442    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" event={"ID":"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff","Type":"ContainerStarted","Data":"f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1"}
Oct 31 01:46:30.344064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:30.342625    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:46:30.344064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:30.343109    5293 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:46:30.344064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:30.343152    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:46:31.336505 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:31.336465    5293 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:46:31.336985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:31.336545    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:46:32.336744 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:32.336700    5293 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:46:32.337267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:32.336769    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:46:33.338138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:33.338099    5293 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:46:33.338644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:33.338170    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:46:34.346300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:34.346262    5293 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:46:34.348687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:34.348646    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:46:35.107616 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:35.107573    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"]
Oct 31 01:46:35.349797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:35.349764    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:46:35.350653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:35.350624    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:46:35.353445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:35.353423    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:46:37.241274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.241239    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"]
Oct 31 01:46:37.242429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.242407    5293 cpu_manager.go:414] "containerMap: removing container" podUID="500852e7-8968-4874-9992-6086ef6e3755" containerName="revoke-keys"
Oct 31 01:46:37.242500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.242432    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500852e7-8968-4874-9992-6086ef6e3755" containerName="revoke-keys"
Oct 31 01:46:37.242615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.242603    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="500852e7-8968-4874-9992-6086ef6e3755" containerName="revoke-keys"
Oct 31 01:46:37.246912 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.246888    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.250995 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.250966    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:46:37.250995 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.250976    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:46:37.251208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.251174    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:46:37.251274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.251249    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:46:37.252925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.252905    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"]
Oct 31 01:46:37.341669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.341632    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxfq\" (UniqueName: \"kubernetes.io/projected/5ebdea7f-71bd-4895-9ca1-8df1133591db-kube-api-access-mpxfq\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.341838 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.341802    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.341928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.341910    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.445382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.445247    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxfq\" (UniqueName: \"kubernetes.io/projected/5ebdea7f-71bd-4895-9ca1-8df1133591db-kube-api-access-mpxfq\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.445756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.445732    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.445959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.445944    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.448327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.448305    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.468064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.454955    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.482086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.478543    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxfq\" (UniqueName: \"kubernetes.io/projected/5ebdea7f-71bd-4895-9ca1-8df1133591db-kube-api-access-mpxfq\") pod \"maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.563319 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.563228    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:37.854691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:37.854658    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"]
Oct 31 01:46:37.854829 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:37.854795    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebdea7f_71bd_4895_9ca1_8df1133591db.slice/crio-0939a3ecebb95d72986a9f90eac3178c96190497606bff872ebacb6d0d7ad80a WatchSource:0}: Error finding container 0939a3ecebb95d72986a9f90eac3178c96190497606bff872ebacb6d0d7ad80a: Status 404 returned error can't find the container with id 0939a3ecebb95d72986a9f90eac3178c96190497606bff872ebacb6d0d7ad80a
Oct 31 01:46:38.111166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:38.111076    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p" event={"ID":"5ebdea7f-71bd-4895-9ca1-8df1133591db","Type":"ContainerStarted","Data":"0939a3ecebb95d72986a9f90eac3178c96190497606bff872ebacb6d0d7ad80a"}
Oct 31 01:46:39.131579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:39.131545    5293 generic.go:356] "Generic (PLEG): container finished" podID="5ebdea7f-71bd-4895-9ca1-8df1133591db" containerID="64a298fd66765d907ab07b1b290da74b7b7f048946cd250c11b5b31d9b62a6d4" exitCode=0
Oct 31 01:46:39.132023 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:39.131834    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p" event={"ID":"5ebdea7f-71bd-4895-9ca1-8df1133591db","Type":"ContainerDied","Data":"64a298fd66765d907ab07b1b290da74b7b7f048946cd250c11b5b31d9b62a6d4"}
Oct 31 01:46:40.110175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.110128    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr" podUID="d2df0794-543e-4701-bddb-e7b14ae8f8d7" containerName="payload-processing" containerID="cri-o://c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246" gracePeriod=25
Oct 31 01:46:40.355711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.355675    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:40.427231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.427197    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebdea7f-71bd-4895-9ca1-8df1133591db-kube-api-access-mpxfq\" (UniqueName: \"kubernetes.io/projected/5ebdea7f-71bd-4895-9ca1-8df1133591db-kube-api-access-mpxfq\") pod \"5ebdea7f-71bd-4895-9ca1-8df1133591db\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") "
Oct 31 01:46:40.427421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.427317    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-xks-ca\") pod \"5ebdea7f-71bd-4895-9ca1-8df1133591db\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") "
Oct 31 01:46:40.427538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.427468    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-openshift-ca\") pod \"5ebdea7f-71bd-4895-9ca1-8df1133591db\" (UID: \"5ebdea7f-71bd-4895-9ca1-8df1133591db\") "
Oct 31 01:46:40.428562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.428535    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-openshift-ca" pod "5ebdea7f-71bd-4895-9ca1-8df1133591db" (UID: "5ebdea7f-71bd-4895-9ca1-8df1133591db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:46:40.431280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.431252    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-xks-ca" pod "5ebdea7f-71bd-4895-9ca1-8df1133591db" (UID: "5ebdea7f-71bd-4895-9ca1-8df1133591db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:46:40.433341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.433319    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebdea7f-71bd-4895-9ca1-8df1133591db-kube-api-access-mpxfq" pod "5ebdea7f-71bd-4895-9ca1-8df1133591db" (UID: "5ebdea7f-71bd-4895-9ca1-8df1133591db"). InnerVolumeSpecName "kube-api-access-mpxfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:40.435781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.435754    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:46:40.529097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.529063    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2df0794-543e-4701-bddb-e7b14ae8f8d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df0794-543e-4701-bddb-e7b14ae8f8d7-plugins-config-volume\") pod \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\" (UID: \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\") "
Oct 31 01:46:40.529316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.529296    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2df0794-543e-4701-bddb-e7b14ae8f8d7-kube-api-access-5fntz\" (UniqueName: \"kubernetes.io/projected/d2df0794-543e-4701-bddb-e7b14ae8f8d7-kube-api-access-5fntz\") pod \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\" (UID: \"d2df0794-543e-4701-bddb-e7b14ae8f8d7\") "
Oct 31 01:46:40.529517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.529489    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2df0794-543e-4701-bddb-e7b14ae8f8d7-plugins-config-volume" pod "d2df0794-543e-4701-bddb-e7b14ae8f8d7" (UID: "d2df0794-543e-4701-bddb-e7b14ae8f8d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:46:40.530510 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.530487    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df0794-543e-4701-bddb-e7b14ae8f8d7-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:40.530612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.530518    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpxfq\" (UniqueName: \"kubernetes.io/projected/5ebdea7f-71bd-4895-9ca1-8df1133591db-kube-api-access-mpxfq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:40.530612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.530534    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:40.530612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.530550    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebdea7f-71bd-4895-9ca1-8df1133591db-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:40.531845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.531810    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2df0794-543e-4701-bddb-e7b14ae8f8d7-kube-api-access-5fntz" pod "d2df0794-543e-4701-bddb-e7b14ae8f8d7" (UID: "d2df0794-543e-4701-bddb-e7b14ae8f8d7"). InnerVolumeSpecName "kube-api-access-5fntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:40.631525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:40.631495    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fntz\" (UniqueName: \"kubernetes.io/projected/d2df0794-543e-4701-bddb-e7b14ae8f8d7-kube-api-access-5fntz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:41.165655 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.165625    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"
Oct 31 01:46:41.165793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.165629    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p" event={"ID":"5ebdea7f-71bd-4895-9ca1-8df1133591db","Type":"ContainerDied","Data":"0939a3ecebb95d72986a9f90eac3178c96190497606bff872ebacb6d0d7ad80a"}
Oct 31 01:46:41.165793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.165758    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0939a3ecebb95d72986a9f90eac3178c96190497606bff872ebacb6d0d7ad80a"
Oct 31 01:46:41.167345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.167325    5293 generic.go:356] "Generic (PLEG): container finished" podID="d2df0794-543e-4701-bddb-e7b14ae8f8d7" containerID="c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246" exitCode=0
Oct 31 01:46:41.167458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.167356    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"
Oct 31 01:46:41.167458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.167415    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr" event={"ID":"d2df0794-543e-4701-bddb-e7b14ae8f8d7","Type":"ContainerDied","Data":"c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246"}
Oct 31 01:46:41.167537 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.167465    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr" event={"ID":"d2df0794-543e-4701-bddb-e7b14ae8f8d7","Type":"ContainerDied","Data":"7c3c6947e252f1d68bc6b9a8e7a6fb34d1f763d6ef24c6fe1d1499d6c5e76a21"}
Oct 31 01:46:41.167537 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.167498    5293 scope.go:122] "RemoveContainer" containerID="c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246"
Oct 31 01:46:41.179791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.179776    5293 scope.go:122] "RemoveContainer" containerID="c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246"
Oct 31 01:46:41.180089 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:46:41.180065    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246\": container with ID starting with c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246 not found: ID does not exist" containerID="c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246"
Oct 31 01:46:41.180135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.180099    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246"} err="failed to get container status \"c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246\": rpc error: code = NotFound desc = could not find container \"c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246\": container with ID starting with c951a8260f1b99a5171fbc22c4db2790e52b58dde56ad27923b5f9df3111e246 not found: ID does not exist"
Oct 31 01:46:41.230321 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.230287    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"]
Oct 31 01:46:41.290345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.290311    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9cc66e-75ffd9f5f-kbckr"]
Oct 31 01:46:41.327116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.327003    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"]
Oct 31 01:46:41.328271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.328250    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5ebdea7f-71bd-4895-9ca1-8df1133591db" containerName="revoke-keys"
Oct 31 01:46:41.328271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.328270    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebdea7f-71bd-4895-9ca1-8df1133591db" containerName="revoke-keys"
Oct 31 01:46:41.328414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.328320    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d2df0794-543e-4701-bddb-e7b14ae8f8d7" containerName="payload-processing"
Oct 31 01:46:41.328414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.328326    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2df0794-543e-4701-bddb-e7b14ae8f8d7" containerName="payload-processing"
Oct 31 01:46:41.328477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.328418    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2df0794-543e-4701-bddb-e7b14ae8f8d7" containerName="payload-processing"
Oct 31 01:46:41.328477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.328429    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebdea7f-71bd-4895-9ca1-8df1133591db" containerName="revoke-keys"
Oct 31 01:46:41.335336 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.335311    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.336674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.336651    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"]
Oct 31 01:46:41.338727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.338707    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:46:41.338972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.338956    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:46:41.339117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.339101    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:46:41.339203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.339158    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:46:41.447867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.447832    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqxc\" (UniqueName: \"kubernetes.io/projected/fabe9d99-a090-4e04-9f0d-2d064670c347-kube-api-access-gjqxc\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.448262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.447895    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.448262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.448082    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.554676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.554627    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqxc\" (UniqueName: \"kubernetes.io/projected/fabe9d99-a090-4e04-9f0d-2d064670c347-kube-api-access-gjqxc\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.554930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.554909    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.555153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.555133    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.555964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.555943    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.557897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.557872    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.566738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.566715    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqxc\" (UniqueName: \"kubernetes.io/projected/fabe9d99-a090-4e04-9f0d-2d064670c347-kube-api-access-gjqxc\") pod \"maas-api-revoke-sub-f26ad416-hmdrd\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.655417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:41.655382    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:41.789597 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:46:41.789557    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:46:42.035781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:42.035700    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"]
Oct 31 01:46:42.042554 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:42.042525    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabe9d99_a090_4e04_9f0d_2d064670c347.slice/crio-50a140e2f4a4f7433ff7971df4ce2ed6d233a1927f1747bb75a59ccdb2908ce0 WatchSource:0}: Error finding container 50a140e2f4a4f7433ff7971df4ce2ed6d233a1927f1747bb75a59ccdb2908ce0: Status 404 returned error can't find the container with id 50a140e2f4a4f7433ff7971df4ce2ed6d233a1927f1747bb75a59ccdb2908ce0
Oct 31 01:46:42.178169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:42.178138    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2df0794-543e-4701-bddb-e7b14ae8f8d7" path="/var/lib/kubelet/pods/d2df0794-543e-4701-bddb-e7b14ae8f8d7/volumes"
Oct 31 01:46:42.178542 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:42.178515    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd" event={"ID":"fabe9d99-a090-4e04-9f0d-2d064670c347","Type":"ContainerStarted","Data":"50a140e2f4a4f7433ff7971df4ce2ed6d233a1927f1747bb75a59ccdb2908ce0"}
Oct 31 01:46:42.488592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:42.488561    5293 scope.go:122] "RemoveContainer" containerID="fa9b7e9570a55073459d21a2e0236a4dc027ee15e16e176fd3c66306fbb8ee5f"
Oct 31 01:46:42.526642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:42.526578    5293 scope.go:122] "RemoveContainer" containerID="b7b790f2926611ce05d1f8eb71be4b51626121bfcce8ff2c4a44421ec63427dd"
Oct 31 01:46:42.578066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:42.572237    5293 scope.go:122] "RemoveContainer" containerID="b43ed0c247e924dbb18e9ad4f248f75d5af12e7bfe177a6683b132765ab94ce1"
Oct 31 01:46:43.192611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:43.192572    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd" event={"ID":"fabe9d99-a090-4e04-9f0d-2d064670c347","Type":"ContainerStarted","Data":"c562ce3a0577aa6ee108c6f552aee2bc5a82db4b63de2babad54782d4a5acc3a"}
Oct 31 01:46:43.230647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:43.230587    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd" podStartSLOduration=2.230566086 podStartE2EDuration="2.230566086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:43.209759655 +0000 UTC m=+1561.158408041" watchObservedRunningTime="2026-10-31 01:46:43.230566086 +0000 UTC m=+1561.179214481"
Oct 31 01:46:44.222069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:44.221815    5293 generic.go:356] "Generic (PLEG): container finished" podID="fabe9d99-a090-4e04-9f0d-2d064670c347" containerID="c562ce3a0577aa6ee108c6f552aee2bc5a82db4b63de2babad54782d4a5acc3a" exitCode=0
Oct 31 01:46:44.222594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:44.222120    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd" event={"ID":"fabe9d99-a090-4e04-9f0d-2d064670c347","Type":"ContainerDied","Data":"c562ce3a0577aa6ee108c6f552aee2bc5a82db4b63de2babad54782d4a5acc3a"}
Oct 31 01:46:45.639364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.639337    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:45.761683 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.761652    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabe9d99-a090-4e04-9f0d-2d064670c347-kube-api-access-gjqxc\" (UniqueName: \"kubernetes.io/projected/fabe9d99-a090-4e04-9f0d-2d064670c347-kube-api-access-gjqxc\") pod \"fabe9d99-a090-4e04-9f0d-2d064670c347\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") "
Oct 31 01:46:45.761896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.761816    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-xks-ca\") pod \"fabe9d99-a090-4e04-9f0d-2d064670c347\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") "
Oct 31 01:46:45.761896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.761847    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-openshift-ca\") pod \"fabe9d99-a090-4e04-9f0d-2d064670c347\" (UID: \"fabe9d99-a090-4e04-9f0d-2d064670c347\") "
Oct 31 01:46:45.762315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.762285    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-openshift-ca" pod "fabe9d99-a090-4e04-9f0d-2d064670c347" (UID: "fabe9d99-a090-4e04-9f0d-2d064670c347"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:46:45.762415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.762405    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:45.763998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.763974    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabe9d99-a090-4e04-9f0d-2d064670c347-kube-api-access-gjqxc" pod "fabe9d99-a090-4e04-9f0d-2d064670c347" (UID: "fabe9d99-a090-4e04-9f0d-2d064670c347"). InnerVolumeSpecName "kube-api-access-gjqxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:45.764144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.764127    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-xks-ca" pod "fabe9d99-a090-4e04-9f0d-2d064670c347" (UID: "fabe9d99-a090-4e04-9f0d-2d064670c347"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:46:45.863890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.863803    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqxc\" (UniqueName: \"kubernetes.io/projected/fabe9d99-a090-4e04-9f0d-2d064670c347-kube-api-access-gjqxc\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:45.863890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:45.863843    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabe9d99-a090-4e04-9f0d-2d064670c347-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:46.243244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:46.243218    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"
Oct 31 01:46:46.243404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:46.243252    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd" event={"ID":"fabe9d99-a090-4e04-9f0d-2d064670c347","Type":"ContainerDied","Data":"50a140e2f4a4f7433ff7971df4ce2ed6d233a1927f1747bb75a59ccdb2908ce0"}
Oct 31 01:46:46.243404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:46.243298    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50a140e2f4a4f7433ff7971df4ce2ed6d233a1927f1747bb75a59ccdb2908ce0"
Oct 31 01:46:47.433333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:47.433294    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"]
Oct 31 01:46:47.440079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:47.440051    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"]
Oct 31 01:46:48.106066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:48.093114    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"]
Oct 31 01:46:48.117064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:48.116022    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b755ceec-j2dbq"]
Oct 31 01:46:48.203097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:48.194399    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca43a12-5791-41a3-8aea-6602f8ef4aef" path="/var/lib/kubelet/pods/6ca43a12-5791-41a3-8aea-6602f8ef4aef/volumes"
Oct 31 01:46:48.694532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:48.694138    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:46:48.694532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:48.694219    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:46:52.068673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.068632    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"]
Oct 31 01:46:52.130013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.129977    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9cc66e-mps7l"]
Oct 31 01:46:52.235579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.235543    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500852e7-8968-4874-9992-6086ef6e3755" path="/var/lib/kubelet/pods/500852e7-8968-4874-9992-6086ef6e3755/volumes"
Oct 31 01:46:52.446183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.446109    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk" podUID="4b2b24bd-88c9-4d96-be81-4993e0d424a0" containerName="payload-processing" containerID="cri-o://17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679" gracePeriod=25
Oct 31 01:46:52.483227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.447800    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw" podUID="c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce" containerName="payload-pre-processing" containerID="cri-o://0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a" gracePeriod=25
Oct 31 01:46:52.512731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.512705    5293 generic.go:356] "Generic (PLEG): container finished" podID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerID="902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1" exitCode=0
Oct 31 01:46:52.512879 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.512864    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" event={"ID":"12d50b4a-007a-4c9e-9bc6-3f9abe048781","Type":"ContainerDied","Data":"902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1"}
Oct 31 01:46:52.513487 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.513462    5293 scope.go:122] "RemoveContainer" containerID="902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1"
Oct 31 01:46:52.515893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.515856    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:52.515976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.515954    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:52.516059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.516026    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:46:52.917132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.917034    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:46:52.921946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:52.921920    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:46:53.056326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.056237    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-plugins-config-volume\") pod \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\" (UID: \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\") "
Oct 31 01:46:53.056326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.056299    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-kube-api-access-c78ms\" (UniqueName: \"kubernetes.io/projected/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-kube-api-access-c78ms\") pod \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\" (UID: \"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce\") "
Oct 31 01:46:53.056548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.056469    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2b24bd-88c9-4d96-be81-4993e0d424a0-kube-api-access-zxxvw\" (UniqueName: \"kubernetes.io/projected/4b2b24bd-88c9-4d96-be81-4993e0d424a0-kube-api-access-zxxvw\") pod \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\" (UID: \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\") "
Oct 31 01:46:53.056652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.056636    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2b24bd-88c9-4d96-be81-4993e0d424a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2b24bd-88c9-4d96-be81-4993e0d424a0-plugins-config-volume\") pod \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\" (UID: \"4b2b24bd-88c9-4d96-be81-4993e0d424a0\") "
Oct 31 01:46:53.056719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.056632    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-plugins-config-volume" pod "c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce" (UID: "c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:46:53.057260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.057227    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2b24bd-88c9-4d96-be81-4993e0d424a0-plugins-config-volume" pod "4b2b24bd-88c9-4d96-be81-4993e0d424a0" (UID: "4b2b24bd-88c9-4d96-be81-4993e0d424a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:46:53.057599 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.057578    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2b24bd-88c9-4d96-be81-4993e0d424a0-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:53.057680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.057607    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:53.058867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.058809    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2b24bd-88c9-4d96-be81-4993e0d424a0-kube-api-access-zxxvw" pod "4b2b24bd-88c9-4d96-be81-4993e0d424a0" (UID: "4b2b24bd-88c9-4d96-be81-4993e0d424a0"). InnerVolumeSpecName "kube-api-access-zxxvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:53.058867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.058828    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-kube-api-access-c78ms" pod "c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce" (UID: "c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce"). InnerVolumeSpecName "kube-api-access-c78ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:53.165323 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.164563    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxvw\" (UniqueName: \"kubernetes.io/projected/4b2b24bd-88c9-4d96-be81-4993e0d424a0-kube-api-access-zxxvw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:53.166069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.166012    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c78ms\" (UniqueName: \"kubernetes.io/projected/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce-kube-api-access-c78ms\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:53.522390 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.522344    5293 generic.go:356] "Generic (PLEG): container finished" podID="c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce" containerID="0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a" exitCode=0
Oct 31 01:46:53.522602 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.522400    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw" event={"ID":"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce","Type":"ContainerDied","Data":"0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a"}
Oct 31 01:46:53.522602 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.522420    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"
Oct 31 01:46:53.522602 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.522495    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw" event={"ID":"c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce","Type":"ContainerDied","Data":"447dc90e90b235faf5581af8c73c04954c1b4ed439ab9a38d35981131f3a5f63"}
Oct 31 01:46:53.522602 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.522526    5293 scope.go:122] "RemoveContainer" containerID="0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a"
Oct 31 01:46:53.526465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.526440    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" event={"ID":"12d50b4a-007a-4c9e-9bc6-3f9abe048781","Type":"ContainerStarted","Data":"ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca"}
Oct 31 01:46:53.530404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.530377    5293 generic.go:356] "Generic (PLEG): container finished" podID="4b2b24bd-88c9-4d96-be81-4993e0d424a0" containerID="17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679" exitCode=0
Oct 31 01:46:53.530516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.530430    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk" event={"ID":"4b2b24bd-88c9-4d96-be81-4993e0d424a0","Type":"ContainerDied","Data":"17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679"}
Oct 31 01:46:53.530516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.530472    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk" event={"ID":"4b2b24bd-88c9-4d96-be81-4993e0d424a0","Type":"ContainerDied","Data":"0811ae085d6d939af86c1bf4f6dbc856c770765b4536eec4a8d0b2676d8a9e7c"}
Oct 31 01:46:53.530644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.530562    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"
Oct 31 01:46:53.547204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.546493    5293 scope.go:122] "RemoveContainer" containerID="0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a"
Oct 31 01:46:53.547204 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:46:53.546827    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a\": container with ID starting with 0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a not found: ID does not exist" containerID="0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a"
Oct 31 01:46:53.547204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.546862    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a"} err="failed to get container status \"0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a\": rpc error: code = NotFound desc = could not find container \"0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a\": container with ID starting with 0b6b82df8b82acb0209eef6792ff94f1d184b45b87d08818de73760f3a422e0a not found: ID does not exist"
Oct 31 01:46:53.547204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.546885    5293 scope.go:122] "RemoveContainer" containerID="17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679"
Oct 31 01:46:53.560067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.560028    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:46:53.560423 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.560397    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 01:46:53.560532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.560450    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 01:46:53.563218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.563197    5293 scope.go:122] "RemoveContainer" containerID="17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679"
Oct 31 01:46:53.563600 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:46:53.563574    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679\": container with ID starting with 17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679 not found: ID does not exist" containerID="17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679"
Oct 31 01:46:53.563712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.563608    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679"} err="failed to get container status \"17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679\": rpc error: code = NotFound desc = could not find container \"17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679\": container with ID starting with 17cb6f4390065d90348d50b645769ff37045252709b4ac927292f2edba5b3679 not found: ID does not exist"
Oct 31 01:46:53.584340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.584304    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"]
Oct 31 01:46:53.599065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.597282    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b634e-548469c467-llrgw"]
Oct 31 01:46:53.635065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.629132    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"]
Oct 31 01:46:53.649075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:53.649019    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b634e-5754895485-wswdk"]
Oct 31 01:46:54.175055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:54.175014    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2b24bd-88c9-4d96-be81-4993e0d424a0" path="/var/lib/kubelet/pods/4b2b24bd-88c9-4d96-be81-4993e0d424a0/volumes"
Oct 31 01:46:54.175645 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:54.175623    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce" path="/var/lib/kubelet/pods/c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce/volumes"
Oct 31 01:46:54.560573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:54.560493    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 01:46:54.560573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:54.560554    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 01:46:55.569072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:55.560318    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 01:46:55.569072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:55.560386    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 01:46:56.028866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:56.028810    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:46:56.560460 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:56.560427    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 01:46:56.560635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:56.560489    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 01:46:57.116145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.116099    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"]
Oct 31 01:46:57.215080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.211492    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3baee4e4-l6z7j"]
Oct 31 01:46:57.431181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.431145    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"]
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.463315    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fabe9d99-a090-4e04-9f0d-2d064670c347" containerName="revoke-keys"
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.479189    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabe9d99-a090-4e04-9f0d-2d064670c347" containerName="revoke-keys"
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.479290    5293 cpu_manager.go:414] "containerMap: removing container" podUID="4b2b24bd-88c9-4d96-be81-4993e0d424a0" containerName="payload-processing"
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.479300    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2b24bd-88c9-4d96-be81-4993e0d424a0" containerName="payload-processing"
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.479348    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce" containerName="payload-pre-processing"
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.479357    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce" containerName="payload-pre-processing"
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.479657    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51dc2ef-e480-4cbc-8b4a-9b3ebbcc75ce" containerName="payload-pre-processing"
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.479681    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabe9d99-a090-4e04-9f0d-2d064670c347" containerName="revoke-keys"
Oct 31 01:46:57.480105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.479696    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2b24bd-88c9-4d96-be81-4993e0d424a0" containerName="payload-processing"
Oct 31 01:46:57.487437 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.487415    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"]
Oct 31 01:46:57.487668 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.487624    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.493651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.493630    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:46:57.493886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.493871    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:46:57.494107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.494094    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:46:57.494420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.494406    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:46:57.508799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.508768    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6sh7\" (UniqueName: \"kubernetes.io/projected/fe9932f1-b97b-410e-ae06-c2ad32369e80-kube-api-access-j6sh7\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.509191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.509171    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.509384 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.509366    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.560539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.560500    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 01:46:57.560723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.560556    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 01:46:57.625444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.625396    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6sh7\" (UniqueName: \"kubernetes.io/projected/fe9932f1-b97b-410e-ae06-c2ad32369e80-kube-api-access-j6sh7\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.625824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.625805    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.625999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.625982    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.627675 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.627608    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.629798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.629777    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.639366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.639342    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6sh7\" (UniqueName: \"kubernetes.io/projected/fe9932f1-b97b-410e-ae06-c2ad32369e80-kube-api-access-j6sh7\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:57.837610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:57.837522    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:46:58.184556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:58.184526    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca69856c-8787-435a-9b6d-869816da8ffc" path="/var/lib/kubelet/pods/ca69856c-8787-435a-9b6d-869816da8ffc/volumes"
Oct 31 01:46:58.189303 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:46:58.189278    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9932f1_b97b_410e_ae06_c2ad32369e80.slice/crio-67e0c58f2b4e7ad74ef9b4c8289eb1921684918b8269f64e48d39cf3517ab72f WatchSource:0}: Error finding container 67e0c58f2b4e7ad74ef9b4c8289eb1921684918b8269f64e48d39cf3517ab72f: Status 404 returned error can't find the container with id 67e0c58f2b4e7ad74ef9b4c8289eb1921684918b8269f64e48d39cf3517ab72f
Oct 31 01:46:58.191804 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:58.191779    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"]
Oct 31 01:46:58.560808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:58.560723    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 01:46:58.560808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:58.560786    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 01:46:58.709065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:58.687613    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj" event={"ID":"fe9932f1-b97b-410e-ae06-c2ad32369e80","Type":"ContainerStarted","Data":"f7bc424a0e4eb400dbfa1e09f6022ae05405c0b602ed8125520377ea0d73cbdd"}
Oct 31 01:46:58.709065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:58.687682    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj" event={"ID":"fe9932f1-b97b-410e-ae06-c2ad32369e80","Type":"ContainerStarted","Data":"67e0c58f2b4e7ad74ef9b4c8289eb1921684918b8269f64e48d39cf3517ab72f"}
Oct 31 01:46:58.742081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:58.740763    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj" podStartSLOduration=1.740740446 podStartE2EDuration="1.740740446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:58.729349821 +0000 UTC m=+1576.677998214" watchObservedRunningTime="2026-10-31 01:46:58.740740446 +0000 UTC m=+1576.689388838"
Oct 31 01:46:59.562467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:59.561457    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 01:46:59.562467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:59.561519    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 01:46:59.780065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:59.775212    5293 generic.go:356] "Generic (PLEG): container finished" podID="fe9932f1-b97b-410e-ae06-c2ad32369e80" containerID="f7bc424a0e4eb400dbfa1e09f6022ae05405c0b602ed8125520377ea0d73cbdd" exitCode=0
Oct 31 01:46:59.780065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:59.775386    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj" event={"ID":"fe9932f1-b97b-410e-ae06-c2ad32369e80","Type":"ContainerDied","Data":"f7bc424a0e4eb400dbfa1e09f6022ae05405c0b602ed8125520377ea0d73cbdd"}
Oct 31 01:46:59.927469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:59.925972    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"]
Oct 31 01:46:59.927469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:46:59.926230    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy" containerID="cri-o://f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1" gracePeriod=30
Oct 31 01:47:00.604066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:00.601665    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:47:00.604066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:00.603630    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:47:00.704066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:00.702757    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:47:00.742078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:00.737177    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"]
Oct 31 01:47:00.870823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:00.868507    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b634e-j8n8p"]
Oct 31 01:47:00.887733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:00.887699    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"]
Oct 31 01:47:00.926205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:00.926178    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"]
Oct 31 01:47:00.926526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:00.926509    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.036364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.035839    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.036364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.035990    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.036364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.036119    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69dt\" (UniqueName: \"kubernetes.io/projected/a59df9bc-dff5-405c-b769-a8d6e18d89d4-kube-api-access-q69dt\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.109758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.109729    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:47:01.138100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.137987    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69dt\" (UniqueName: \"kubernetes.io/projected/a59df9bc-dff5-405c-b769-a8d6e18d89d4-kube-api-access-q69dt\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.138293 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.138272    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.138434 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.138416    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.139283 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.139205    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.141439 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.141418    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.149481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.149455    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69dt\" (UniqueName: \"kubernetes.io/projected/a59df9bc-dff5-405c-b769-a8d6e18d89d4-kube-api-access-q69dt\") pod \"maas-api-revoke-keys-e2e-api-b-b6b833-w26g2\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.240180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.239347    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-xks-ca\") pod \"fe9932f1-b97b-410e-ae06-c2ad32369e80\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") "
Oct 31 01:47:01.240180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.239415    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-openshift-ca\") pod \"fe9932f1-b97b-410e-ae06-c2ad32369e80\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") "
Oct 31 01:47:01.240180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.239717    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9932f1-b97b-410e-ae06-c2ad32369e80-kube-api-access-j6sh7\" (UniqueName: \"kubernetes.io/projected/fe9932f1-b97b-410e-ae06-c2ad32369e80-kube-api-access-j6sh7\") pod \"fe9932f1-b97b-410e-ae06-c2ad32369e80\" (UID: \"fe9932f1-b97b-410e-ae06-c2ad32369e80\") "
Oct 31 01:47:01.240512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.240445    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-openshift-ca" pod "fe9932f1-b97b-410e-ae06-c2ad32369e80" (UID: "fe9932f1-b97b-410e-ae06-c2ad32369e80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:01.243093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.243010    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9932f1-b97b-410e-ae06-c2ad32369e80-kube-api-access-j6sh7" pod "fe9932f1-b97b-410e-ae06-c2ad32369e80" (UID: "fe9932f1-b97b-410e-ae06-c2ad32369e80"). InnerVolumeSpecName "kube-api-access-j6sh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:01.243221 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.243164    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-xks-ca" pod "fe9932f1-b97b-410e-ae06-c2ad32369e80" (UID: "fe9932f1-b97b-410e-ae06-c2ad32369e80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:47:01.254560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.254536    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:01.342442 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.342416    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:01.342597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.342445    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9932f1-b97b-410e-ae06-c2ad32369e80-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:01.342597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.342462    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6sh7\" (UniqueName: \"kubernetes.io/projected/fe9932f1-b97b-410e-ae06-c2ad32369e80-kube-api-access-j6sh7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:01.578829 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:01.578798    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59df9bc_dff5_405c_b769_a8d6e18d89d4.slice/crio-9dddae65aa0b8d5bd792235e46619b3a8f9ae26adad4670e3c7143e603ceb6da WatchSource:0}: Error finding container 9dddae65aa0b8d5bd792235e46619b3a8f9ae26adad4670e3c7143e603ceb6da: Status 404 returned error can't find the container with id 9dddae65aa0b8d5bd792235e46619b3a8f9ae26adad4670e3c7143e603ceb6da
Oct 31 01:47:01.578986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.578967    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"]
Oct 31 01:47:01.876821 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.876737    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"
Oct 31 01:47:01.877313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.876800    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj" event={"ID":"fe9932f1-b97b-410e-ae06-c2ad32369e80","Type":"ContainerDied","Data":"67e0c58f2b4e7ad74ef9b4c8289eb1921684918b8269f64e48d39cf3517ab72f"}
Oct 31 01:47:01.877313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.876863    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67e0c58f2b4e7ad74ef9b4c8289eb1921684918b8269f64e48d39cf3517ab72f"
Oct 31 01:47:01.879418 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:01.879393    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2" event={"ID":"a59df9bc-dff5-405c-b769-a8d6e18d89d4","Type":"ContainerStarted","Data":"9dddae65aa0b8d5bd792235e46619b3a8f9ae26adad4670e3c7143e603ceb6da"}
Oct 31 01:47:02.030188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:02.030143    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"]
Oct 31 01:47:02.035696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:02.035664    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b8328a-ddpzf"]
Oct 31 01:47:02.282441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:02.282413    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c94eb49-dffe-47d8-8777-f1331e88b016" path="/var/lib/kubelet/pods/1c94eb49-dffe-47d8-8777-f1331e88b016/volumes"
Oct 31 01:47:02.283306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:02.283283    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ebdea7f-71bd-4895-9ca1-8df1133591db" path="/var/lib/kubelet/pods/5ebdea7f-71bd-4895-9ca1-8df1133591db/volumes"
Oct 31 01:47:02.888444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:02.888410    5293 generic.go:356] "Generic (PLEG): container finished" podID="a59df9bc-dff5-405c-b769-a8d6e18d89d4" containerID="4bac1898275522dfac13017733a492875827dfc006cdc16ca584ee64fabb69af" exitCode=0
Oct 31 01:47:02.888897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:02.888452    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2" event={"ID":"a59df9bc-dff5-405c-b769-a8d6e18d89d4","Type":"ContainerDied","Data":"4bac1898275522dfac13017733a492875827dfc006cdc16ca584ee64fabb69af"}
Oct 31 01:47:04.035916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.035883    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"]
Oct 31 01:47:04.043665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.043622    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"]
Oct 31 01:47:04.048369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.048330    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f2f42e-6dgtt"]
Oct 31 01:47:04.052058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.052015    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5259240-5chbr"]
Oct 31 01:47:04.065743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.065719    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:04.119109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.100796    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59df9bc-dff5-405c-b769-a8d6e18d89d4-kube-api-access-q69dt\" (UniqueName: \"kubernetes.io/projected/a59df9bc-dff5-405c-b769-a8d6e18d89d4-kube-api-access-q69dt\") pod \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") "
Oct 31 01:47:04.119109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.100846    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-xks-ca\") pod \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") "
Oct 31 01:47:04.119109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.100946    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-openshift-ca\") pod \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\" (UID: \"a59df9bc-dff5-405c-b769-a8d6e18d89d4\") "
Oct 31 01:47:04.119109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.102281    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-openshift-ca" pod "a59df9bc-dff5-405c-b769-a8d6e18d89d4" (UID: "a59df9bc-dff5-405c-b769-a8d6e18d89d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:04.119109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.105193    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-xks-ca" pod "a59df9bc-dff5-405c-b769-a8d6e18d89d4" (UID: "a59df9bc-dff5-405c-b769-a8d6e18d89d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:47:04.119109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.105204    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59df9bc-dff5-405c-b769-a8d6e18d89d4-kube-api-access-q69dt" pod "a59df9bc-dff5-405c-b769-a8d6e18d89d4" (UID: "a59df9bc-dff5-405c-b769-a8d6e18d89d4"). InnerVolumeSpecName "kube-api-access-q69dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:04.177084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.176978    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6be199-c533-4e82-b569-59bd860268b8" path="/var/lib/kubelet/pods/0f6be199-c533-4e82-b569-59bd860268b8/volumes"
Oct 31 01:47:04.177652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.177628    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80f96a4-dabc-4ac6-a225-76e1f3f7f860" path="/var/lib/kubelet/pods/d80f96a4-dabc-4ac6-a225-76e1f3f7f860/volumes"
Oct 31 01:47:04.204072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.204027    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q69dt\" (UniqueName: \"kubernetes.io/projected/a59df9bc-dff5-405c-b769-a8d6e18d89d4-kube-api-access-q69dt\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:04.204252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.204234    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:04.204318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.204258    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59df9bc-dff5-405c-b769-a8d6e18d89d4-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:04.910682 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.910651    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"
Oct 31 01:47:04.910886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.910659    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2" event={"ID":"a59df9bc-dff5-405c-b769-a8d6e18d89d4","Type":"ContainerDied","Data":"9dddae65aa0b8d5bd792235e46619b3a8f9ae26adad4670e3c7143e603ceb6da"}
Oct 31 01:47:04.910886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:04.910756    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dddae65aa0b8d5bd792235e46619b3a8f9ae26adad4670e3c7143e603ceb6da"
Oct 31 01:47:05.336514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.336307    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:47:05.445173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445123    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-certs\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.445415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445233    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-envoy\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.445415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445324    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-credential-socket\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.445415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445355    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-socket\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.445680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445426    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-kube-api-access-q4vdh\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-kube-api-access-q4vdh\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.445680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445458    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-token\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-token\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.445680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445502    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-data\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.445680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445606    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istiod-ca-cert\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.445999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.445751    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-podinfo\") pod \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\" (UID: \"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff\") "
Oct 31 01:47:05.472064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.447515    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-data" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.472064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.447761    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-socket" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.472064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.447836    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istiod-ca-cert" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:05.472064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.447999    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-certs" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.472064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.449570    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-credential-socket" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.518276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.513262    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-kube-api-access-q4vdh" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "kube-api-access-q4vdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:05.518276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.516530    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-token" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:05.554125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.553472    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vdh\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-kube-api-access-q4vdh\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.554125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.553511    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.554125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.553526    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.554125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.553557    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.554125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.553574    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.554125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.553588    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.554125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.553602    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.739379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.739343    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-podinfo" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:05.769206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.757427    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.771513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.771127    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-envoy" pod "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" (UID: "3829e5be-ba1e-44ea-8d3a-42e444e6c2ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.878202 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.872692    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.974444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.974405    5293 generic.go:356] "Generic (PLEG): container finished" podID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerID="f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1" exitCode=0
Oct 31 01:47:05.974642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.974530    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" event={"ID":"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff","Type":"ContainerDied","Data":"f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1"}
Oct 31 01:47:05.974642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.974614    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c" event={"ID":"3829e5be-ba1e-44ea-8d3a-42e444e6c2ff","Type":"ContainerDied","Data":"13be72c92facfcc1bb9c68b766d08385e9a77752d161a0f2f64757083c2e8b66"}
Oct 31 01:47:05.974768 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.974642    5293 scope.go:122] "RemoveContainer" containerID="f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1"
Oct 31 01:47:05.974841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.974828    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"
Oct 31 01:47:05.991875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:05.991833    5293 scope.go:122] "RemoveContainer" containerID="a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd"
Oct 31 01:47:06.004923 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:06.004881    5293 scope.go:122] "RemoveContainer" containerID="f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1"
Oct 31 01:47:06.005283 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:06.005258    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1\": container with ID starting with f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1 not found: ID does not exist" containerID="f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1"
Oct 31 01:47:06.005362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:06.005297    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1"} err="failed to get container status \"f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1\": rpc error: code = NotFound desc = could not find container \"f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1\": container with ID starting with f37bded0c88799d0392afca25fd115cb6cfb684ffc0a84423acaecc3d73860e1 not found: ID does not exist"
Oct 31 01:47:06.005362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:06.005320    5293 scope.go:122] "RemoveContainer" containerID="a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd"
Oct 31 01:47:06.005636 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:06.005614    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd\": container with ID starting with a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd not found: ID does not exist" containerID="a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd"
Oct 31 01:47:06.005689 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:06.005645    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd"} err="failed to get container status \"a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd\": rpc error: code = NotFound desc = could not find container \"a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd\": container with ID starting with a93ace96f6464d2e23a85948fa2db5365ae67094f02c6671135231fa6c660bbd not found: ID does not exist"
Oct 31 01:47:06.008660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:06.008638    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"]
Oct 31 01:47:06.013001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:06.012977    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-rnm6c"]
Oct 31 01:47:06.176227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:06.176188    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" path="/var/lib/kubelet/pods/3829e5be-ba1e-44ea-8d3a-42e444e6c2ff/volumes"
Oct 31 01:47:08.280368 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.279754    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"]
Oct 31 01:47:08.280368 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.280003    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy" containerID="cri-o://5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb" gracePeriod=30
Oct 31 01:47:08.289934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.289724    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"]
Oct 31 01:47:08.290097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.289950    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy" containerID="cri-o://07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401" gracePeriod=30
Oct 31 01:47:08.790692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.790657    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"]
Oct 31 01:47:08.791444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791422    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fe9932f1-b97b-410e-ae06-c2ad32369e80" containerName="revoke-keys"
Oct 31 01:47:08.791575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791445    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9932f1-b97b-410e-ae06-c2ad32369e80" containerName="revoke-keys"
Oct 31 01:47:08.791575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791489    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a59df9bc-dff5-405c-b769-a8d6e18d89d4" containerName="revoke-keys"
Oct 31 01:47:08.791575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791496    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59df9bc-dff5-405c-b769-a8d6e18d89d4" containerName="revoke-keys"
Oct 31 01:47:08.791575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791511    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy"
Oct 31 01:47:08.791575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791519    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy"
Oct 31 01:47:08.791575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791536    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy"
Oct 31 01:47:08.791575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791542    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy"
Oct 31 01:47:08.791963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791750    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9932f1-b97b-410e-ae06-c2ad32369e80" containerName="revoke-keys"
Oct 31 01:47:08.791963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791773    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59df9bc-dff5-405c-b769-a8d6e18d89d4" containerName="revoke-keys"
Oct 31 01:47:08.791963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791783    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy"
Oct 31 01:47:08.791963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.791795    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3829e5be-ba1e-44ea-8d3a-42e444e6c2ff" containerName="istio-proxy"
Oct 31 01:47:08.796852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.796831    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.803617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.800398    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:47:08.803617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.800473    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:47:08.803617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.803565    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:47:08.803865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.803699    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:47:08.806610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.806581    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"]
Oct 31 01:47:08.857713 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.857573    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4dd\" (UniqueName: \"kubernetes.io/projected/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-kube-api-access-cp4dd\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.858492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.857972    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.858492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.858097    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.890104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.890029    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"]
Oct 31 01:47:08.894784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.894763    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:08.906214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.906134    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"]
Oct 31 01:47:08.959625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.959593    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshpd\" (UniqueName: \"kubernetes.io/projected/417ee2e3-c6e4-4ba9-933b-d57ba7457454-kube-api-access-bshpd\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:08.959772 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.959753    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:08.959922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.959905    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4dd\" (UniqueName: \"kubernetes.io/projected/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-kube-api-access-cp4dd\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.959982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.959948    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:08.960019 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.959987    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.960170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.960149    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.960971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.960945    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.962748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.962729    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:08.971843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:08.971820    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4dd\" (UniqueName: \"kubernetes.io/projected/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-kube-api-access-cp4dd\") pod \"maas-api-revoke-keys-e2e-api-b-0239db-t7qrt\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:09.061795 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.061714    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshpd\" (UniqueName: \"kubernetes.io/projected/417ee2e3-c6e4-4ba9-933b-d57ba7457454-kube-api-access-bshpd\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:09.061962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.061833    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:09.061962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.061906    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:09.062591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.062566    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:09.064611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.064579    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:09.074301 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.074273    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshpd\" (UniqueName: \"kubernetes.io/projected/417ee2e3-c6e4-4ba9-933b-d57ba7457454-kube-api-access-bshpd\") pod \"maas-api-revoke-sub-edd9cdb0-p8f8g\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:09.115348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.115318    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:09.213913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.213690    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:09.402105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.402075    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"]
Oct 31 01:47:09.402837 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:09.402809    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc650adc_eaf9_41e1_a79a_0d9e23f09b79.slice/crio-dcd67b502466bf065b96e3fc263f99a44d06741cf59862586c13cc506c739ee8 WatchSource:0}: Error finding container dcd67b502466bf065b96e3fc263f99a44d06741cf59862586c13cc506c739ee8: Status 404 returned error can't find the container with id dcd67b502466bf065b96e3fc263f99a44d06741cf59862586c13cc506c739ee8
Oct 31 01:47:09.514753 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:09.514722    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417ee2e3_c6e4_4ba9_933b_d57ba7457454.slice/crio-f091b718a7aa6209543287011f5a5356b141355d69d9a51b3603363c8713d0c1 WatchSource:0}: Error finding container f091b718a7aa6209543287011f5a5356b141355d69d9a51b3603363c8713d0c1: Status 404 returned error can't find the container with id f091b718a7aa6209543287011f5a5356b141355d69d9a51b3603363c8713d0c1
Oct 31 01:47:09.514886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:09.514862    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"]
Oct 31 01:47:10.040405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:10.040294    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt" event={"ID":"cc650adc-eaf9-41e1-a79a-0d9e23f09b79","Type":"ContainerStarted","Data":"d76cbef5f4b865949226bc6ab2f70b1d48fb4bf30d9a91f4d80e9a2f32d168f7"}
Oct 31 01:47:10.040405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:10.040390    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt" event={"ID":"cc650adc-eaf9-41e1-a79a-0d9e23f09b79","Type":"ContainerStarted","Data":"dcd67b502466bf065b96e3fc263f99a44d06741cf59862586c13cc506c739ee8"}
Oct 31 01:47:10.041990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:10.041961    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g" event={"ID":"417ee2e3-c6e4-4ba9-933b-d57ba7457454","Type":"ContainerStarted","Data":"8428f52e3630cb86be2a7f4fa0571625c1659c8896716b930a406ca0c6964888"}
Oct 31 01:47:10.042130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:10.042006    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g" event={"ID":"417ee2e3-c6e4-4ba9-933b-d57ba7457454","Type":"ContainerStarted","Data":"f091b718a7aa6209543287011f5a5356b141355d69d9a51b3603363c8713d0c1"}
Oct 31 01:47:10.058872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:10.058826    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt" podStartSLOduration=2.05881302 podStartE2EDuration="2.05881302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:10.056811179 +0000 UTC m=+1588.005459570" watchObservedRunningTime="2026-10-31 01:47:10.05881302 +0000 UTC m=+1588.007461412"
Oct 31 01:47:10.083453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:10.082782    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g" podStartSLOduration=2.082765316 podStartE2EDuration="2.082765316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:10.075425307 +0000 UTC m=+1588.024073699" watchObservedRunningTime="2026-10-31 01:47:10.082765316 +0000 UTC m=+1588.031413713"
Oct 31 01:47:10.897924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:10.897890    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"]
Oct 31 01:47:10.913179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:10.913153    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"]
Oct 31 01:47:11.059113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:11.059081    5293 generic.go:356] "Generic (PLEG): container finished" podID="cc650adc-eaf9-41e1-a79a-0d9e23f09b79" containerID="d76cbef5f4b865949226bc6ab2f70b1d48fb4bf30d9a91f4d80e9a2f32d168f7" exitCode=0
Oct 31 01:47:11.059277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:11.059151    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt" event={"ID":"cc650adc-eaf9-41e1-a79a-0d9e23f09b79","Type":"ContainerDied","Data":"d76cbef5f4b865949226bc6ab2f70b1d48fb4bf30d9a91f4d80e9a2f32d168f7"}
Oct 31 01:47:11.060888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:11.060860    5293 generic.go:356] "Generic (PLEG): container finished" podID="417ee2e3-c6e4-4ba9-933b-d57ba7457454" containerID="8428f52e3630cb86be2a7f4fa0571625c1659c8896716b930a406ca0c6964888" exitCode=0
Oct 31 01:47:11.061013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:11.060888    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g" event={"ID":"417ee2e3-c6e4-4ba9-933b-d57ba7457454","Type":"ContainerDied","Data":"8428f52e3630cb86be2a7f4fa0571625c1659c8896716b930a406ca0c6964888"}
Oct 31 01:47:11.780908 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:11.780869    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:47:12.300107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.300079    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:12.303582 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.303553    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:12.346272 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.346176    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417ee2e3-c6e4-4ba9-933b-d57ba7457454-kube-api-access-bshpd\" (UniqueName: \"kubernetes.io/projected/417ee2e3-c6e4-4ba9-933b-d57ba7457454-kube-api-access-bshpd\") pod \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") "
Oct 31 01:47:12.346476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.346287    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-xks-ca\") pod \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") "
Oct 31 01:47:12.346476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.346367    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-openshift-ca\") pod \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\" (UID: \"417ee2e3-c6e4-4ba9-933b-d57ba7457454\") "
Oct 31 01:47:12.351249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.347826    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-openshift-ca" pod "417ee2e3-c6e4-4ba9-933b-d57ba7457454" (UID: "417ee2e3-c6e4-4ba9-933b-d57ba7457454"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:12.351249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.350993    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417ee2e3-c6e4-4ba9-933b-d57ba7457454-kube-api-access-bshpd" pod "417ee2e3-c6e4-4ba9-933b-d57ba7457454" (UID: "417ee2e3-c6e4-4ba9-933b-d57ba7457454"). InnerVolumeSpecName "kube-api-access-bshpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:12.351249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.351142    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-xks-ca" pod "417ee2e3-c6e4-4ba9-933b-d57ba7457454" (UID: "417ee2e3-c6e4-4ba9-933b-d57ba7457454"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:47:12.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.468769    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-kube-api-access-cp4dd\" (UniqueName: \"kubernetes.io/projected/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-kube-api-access-cp4dd\") pod \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") "
Oct 31 01:47:12.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.468880    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-xks-ca\") pod \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") "
Oct 31 01:47:12.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.468969    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-openshift-ca\") pod \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\" (UID: \"cc650adc-eaf9-41e1-a79a-0d9e23f09b79\") "
Oct 31 01:47:12.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.469871    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bshpd\" (UniqueName: \"kubernetes.io/projected/417ee2e3-c6e4-4ba9-933b-d57ba7457454-kube-api-access-bshpd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:12.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.469900    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:12.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.469915    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417ee2e3-c6e4-4ba9-933b-d57ba7457454-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:12.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.470355    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-openshift-ca" pod "cc650adc-eaf9-41e1-a79a-0d9e23f09b79" (UID: "cc650adc-eaf9-41e1-a79a-0d9e23f09b79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:12.506102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.504221    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-kube-api-access-cp4dd" pod "cc650adc-eaf9-41e1-a79a-0d9e23f09b79" (UID: "cc650adc-eaf9-41e1-a79a-0d9e23f09b79"). InnerVolumeSpecName "kube-api-access-cp4dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:12.526534 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.526103    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:47:12.526534 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.526192    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:47:12.535488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.534110    5293 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:47:12.535488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.534179    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:47:12.556825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.556442    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-xks-ca" pod "cc650adc-eaf9-41e1-a79a-0d9e23f09b79" (UID: "cc650adc-eaf9-41e1-a79a-0d9e23f09b79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:47:12.571799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.571771    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4dd\" (UniqueName: \"kubernetes.io/projected/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-kube-api-access-cp4dd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:12.571799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.571801    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:12.571989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:12.571814    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc650adc-eaf9-41e1-a79a-0d9e23f09b79-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.079688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.079658    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"
Oct 31 01:47:13.079850 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.079684    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt" event={"ID":"cc650adc-eaf9-41e1-a79a-0d9e23f09b79","Type":"ContainerDied","Data":"dcd67b502466bf065b96e3fc263f99a44d06741cf59862586c13cc506c739ee8"}
Oct 31 01:47:13.079850 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.079775    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcd67b502466bf065b96e3fc263f99a44d06741cf59862586c13cc506c739ee8"
Oct 31 01:47:13.081544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.081517    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"
Oct 31 01:47:13.081737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.081572    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g" event={"ID":"417ee2e3-c6e4-4ba9-933b-d57ba7457454","Type":"ContainerDied","Data":"f091b718a7aa6209543287011f5a5356b141355d69d9a51b3603363c8713d0c1"}
Oct 31 01:47:13.081737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.081624    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f091b718a7aa6209543287011f5a5356b141355d69d9a51b3603363c8713d0c1"
Oct 31 01:47:13.335892 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:13.335833    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e99c30_f334_44be_b22b_a1e12acd6762.slice/crio-5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df3b6eb_f9da_446a_a551_3c56f0a3615c.slice/crio-conmon-07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:47:13.628926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.628889    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:47:13.662672 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.662645    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:47:13.792787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.792749    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-data\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.792976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.792824    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-credential-socket\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.792976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.792854    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-certs\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.792976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.792944    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istiod-ca-cert\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.792976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.792961    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-podinfo\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.793197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.792988    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-socket\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.793197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793027    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-podinfo\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.793197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793115    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-kube-api-access-p7q5w\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-kube-api-access-p7q5w\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.793197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793159    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-certs\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.793379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793202    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-credential-socket\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.793379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793113    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-data" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.793379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793241    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-envoy\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.793379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793210    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-credential-socket" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.793379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793312    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-token\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-token\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.793379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793372    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-envoy\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793350    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-socket" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793402    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-certs" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793440    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e99c30-f334-44be-b22b-a1e12acd6762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e99c30-f334-44be-b22b-a1e12acd6762-istiod-ca-cert\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793455    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istiod-ca-cert" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793477    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-token\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-token\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793505    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-kube-api-access-2cpcg\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-kube-api-access-2cpcg\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793527    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-certs" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793562    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-data\") pod \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\" (UID: \"5df3b6eb-f9da-446a-a551-3c56f0a3615c\") "
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793629    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-socket\") pod \"c7e99c30-f334-44be-b22b-a1e12acd6762\" (UID: \"c7e99c30-f334-44be-b22b-a1e12acd6762\") "
Oct 31 01:47:13.793669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793637    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-credential-socket" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.794352 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.793829    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e99c30-f334-44be-b22b-a1e12acd6762-istiod-ca-cert" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:13.794448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794421    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.794511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794453    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.794511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794470    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.794511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794483    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.794511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794497    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e99c30-f334-44be-b22b-a1e12acd6762-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.794982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794510    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.794982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794525    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.794982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794539    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.794982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794759    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-socket" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.794982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.794936    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-data" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.796632 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.796566    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-token" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:13.796632 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.796591    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-podinfo" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:13.796834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.796815    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-envoy" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.796998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.796973    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-kube-api-access-2cpcg" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "kube-api-access-2cpcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:13.797254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.797216    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-podinfo" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:13.797476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.797456    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-kube-api-access-p7q5w" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "kube-api-access-p7q5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:13.797604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.797575    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-envoy" pod "c7e99c30-f334-44be-b22b-a1e12acd6762" (UID: "c7e99c30-f334-44be-b22b-a1e12acd6762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.797604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.797577    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-token" pod "5df3b6eb-f9da-446a-a551-3c56f0a3615c" (UID: "5df3b6eb-f9da-446a-a551-3c56f0a3615c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896650    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896681    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896696    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7q5w\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-kube-api-access-p7q5w\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896710    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896724    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e99c30-f334-44be-b22b-a1e12acd6762-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896739    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896752    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896765    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cpcg\" (UniqueName: \"kubernetes.io/projected/5df3b6eb-f9da-446a-a551-3c56f0a3615c-kube-api-access-2cpcg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896779    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df3b6eb-f9da-446a-a551-3c56f0a3615c-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.897121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:13.896791    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e99c30-f334-44be-b22b-a1e12acd6762-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:14.090707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.090674    5293 generic.go:356] "Generic (PLEG): container finished" podID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerID="5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb" exitCode=0
Oct 31 01:47:14.090871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.090721    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"
Oct 31 01:47:14.090871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.090762    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" event={"ID":"c7e99c30-f334-44be-b22b-a1e12acd6762","Type":"ContainerDied","Data":"5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb"}
Oct 31 01:47:14.090871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.090819    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5" event={"ID":"c7e99c30-f334-44be-b22b-a1e12acd6762","Type":"ContainerDied","Data":"e77dbd32ac148e768ca95001a3fd4878ba061900abc9b400b4807057fe212e50"}
Oct 31 01:47:14.090871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.090854    5293 scope.go:122] "RemoveContainer" containerID="5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb"
Oct 31 01:47:14.092514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.092491    5293 generic.go:356] "Generic (PLEG): container finished" podID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerID="07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401" exitCode=0
Oct 31 01:47:14.092630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.092601    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" event={"ID":"5df3b6eb-f9da-446a-a551-3c56f0a3615c","Type":"ContainerDied","Data":"07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401"}
Oct 31 01:47:14.092696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.092655    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29" event={"ID":"5df3b6eb-f9da-446a-a551-3c56f0a3615c","Type":"ContainerDied","Data":"b77abe1fbf95f4134136c49d726f57d5ac0956ab14d1e39b2bd05efc5f26bcb8"}
Oct 31 01:47:14.092696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.092687    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"
Oct 31 01:47:14.105944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.105925    5293 scope.go:122] "RemoveContainer" containerID="5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb"
Oct 31 01:47:14.106326 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:14.106306    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb\": container with ID starting with 5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb not found: ID does not exist" containerID="5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb"
Oct 31 01:47:14.106409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.106337    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb"} err="failed to get container status \"5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb\": rpc error: code = NotFound desc = could not find container \"5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb\": container with ID starting with 5a87e36c1c10d9a03997239c38b6c5e285ab607c7bd974d1e10a43785e82fcfb not found: ID does not exist"
Oct 31 01:47:14.106409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.106361    5293 scope.go:122] "RemoveContainer" containerID="07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401"
Oct 31 01:47:14.120153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.120135    5293 scope.go:122] "RemoveContainer" containerID="07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401"
Oct 31 01:47:14.120238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.120224    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"]
Oct 31 01:47:14.120461 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:14.120428    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401\": container with ID starting with 07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401 not found: ID does not exist" containerID="07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401"
Oct 31 01:47:14.120518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.120472    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401"} err="failed to get container status \"07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401\": rpc error: code = NotFound desc = could not find container \"07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401\": container with ID starting with 07ccc4e065b32206aaebad02b4a784dc4344c694c2aca44a5bf68ad24a190401 not found: ID does not exist"
Oct 31 01:47:14.125553 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.125534    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-j65c5"]
Oct 31 01:47:14.144289 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.144257    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"]
Oct 31 01:47:14.148922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.148855    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-z5f29"]
Oct 31 01:47:14.175758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.175725    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" path="/var/lib/kubelet/pods/5df3b6eb-f9da-446a-a551-3c56f0a3615c/volumes"
Oct 31 01:47:14.176461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:14.176447    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" path="/var/lib/kubelet/pods/c7e99c30-f334-44be-b22b-a1e12acd6762/volumes"
Oct 31 01:47:15.899109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:15.899066    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st" podUID="b801913c-b11c-49f6-9e07-7cf83fb28d7f" containerName="payload-processing" containerID="cri-o://66b1e5e165ff79f9c7d5280c352a7d0d241a1c299b943e97fee0455e9c0f027f" gracePeriod=25
Oct 31 01:47:15.914129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:15.914089    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9" podUID="07c887a3-622c-4808-b16a-c0b5d359f3fd" containerName="payload-pre-processing" containerID="cri-o://6049b1d4668787da89d71588337e89cd2a1422588007b8a5d63a414503410351" gracePeriod=25
Oct 31 01:47:16.147152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.147112    5293 generic.go:356] "Generic (PLEG): container finished" podID="07c887a3-622c-4808-b16a-c0b5d359f3fd" containerID="6049b1d4668787da89d71588337e89cd2a1422588007b8a5d63a414503410351" exitCode=0
Oct 31 01:47:16.147339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.147315    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9" event={"ID":"07c887a3-622c-4808-b16a-c0b5d359f3fd","Type":"ContainerDied","Data":"6049b1d4668787da89d71588337e89cd2a1422588007b8a5d63a414503410351"}
Oct 31 01:47:16.150945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.150869    5293 generic.go:356] "Generic (PLEG): container finished" podID="b801913c-b11c-49f6-9e07-7cf83fb28d7f" containerID="66b1e5e165ff79f9c7d5280c352a7d0d241a1c299b943e97fee0455e9c0f027f" exitCode=0
Oct 31 01:47:16.151132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.151016    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st" event={"ID":"b801913c-b11c-49f6-9e07-7cf83fb28d7f","Type":"ContainerDied","Data":"66b1e5e165ff79f9c7d5280c352a7d0d241a1c299b943e97fee0455e9c0f027f"}
Oct 31 01:47:16.301605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.301563    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:47:16.357024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.356992    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:47:16.455562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.455466    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b801913c-b11c-49f6-9e07-7cf83fb28d7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b801913c-b11c-49f6-9e07-7cf83fb28d7f-plugins-config-volume\") pod \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\" (UID: \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\") "
Oct 31 01:47:16.455562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.455533    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b801913c-b11c-49f6-9e07-7cf83fb28d7f-kube-api-access-bj6h4\" (UniqueName: \"kubernetes.io/projected/b801913c-b11c-49f6-9e07-7cf83fb28d7f-kube-api-access-bj6h4\") pod \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\" (UID: \"b801913c-b11c-49f6-9e07-7cf83fb28d7f\") "
Oct 31 01:47:16.455806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.455650    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c887a3-622c-4808-b16a-c0b5d359f3fd-kube-api-access-7pjrg\" (UniqueName: \"kubernetes.io/projected/07c887a3-622c-4808-b16a-c0b5d359f3fd-kube-api-access-7pjrg\") pod \"07c887a3-622c-4808-b16a-c0b5d359f3fd\" (UID: \"07c887a3-622c-4808-b16a-c0b5d359f3fd\") "
Oct 31 01:47:16.455806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.455722    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c887a3-622c-4808-b16a-c0b5d359f3fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c887a3-622c-4808-b16a-c0b5d359f3fd-plugins-config-volume\") pod \"07c887a3-622c-4808-b16a-c0b5d359f3fd\" (UID: \"07c887a3-622c-4808-b16a-c0b5d359f3fd\") "
Oct 31 01:47:16.456163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.456138    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c887a3-622c-4808-b16a-c0b5d359f3fd-plugins-config-volume" pod "07c887a3-622c-4808-b16a-c0b5d359f3fd" (UID: "07c887a3-622c-4808-b16a-c0b5d359f3fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:16.456272 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.456256    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b801913c-b11c-49f6-9e07-7cf83fb28d7f-plugins-config-volume" pod "b801913c-b11c-49f6-9e07-7cf83fb28d7f" (UID: "b801913c-b11c-49f6-9e07-7cf83fb28d7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:16.456569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.456550    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b801913c-b11c-49f6-9e07-7cf83fb28d7f-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.456623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.456573    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c887a3-622c-4808-b16a-c0b5d359f3fd-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.457836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.457818    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b801913c-b11c-49f6-9e07-7cf83fb28d7f-kube-api-access-bj6h4" pod "b801913c-b11c-49f6-9e07-7cf83fb28d7f" (UID: "b801913c-b11c-49f6-9e07-7cf83fb28d7f"). InnerVolumeSpecName "kube-api-access-bj6h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:16.458283 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.458266    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c887a3-622c-4808-b16a-c0b5d359f3fd-kube-api-access-7pjrg" pod "07c887a3-622c-4808-b16a-c0b5d359f3fd" (UID: "07c887a3-622c-4808-b16a-c0b5d359f3fd"). InnerVolumeSpecName "kube-api-access-7pjrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:16.558426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.558372    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj6h4\" (UniqueName: \"kubernetes.io/projected/b801913c-b11c-49f6-9e07-7cf83fb28d7f-kube-api-access-bj6h4\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.558426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:16.558422    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjrg\" (UniqueName: \"kubernetes.io/projected/07c887a3-622c-4808-b16a-c0b5d359f3fd-kube-api-access-7pjrg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:17.161334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.161297    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st" event={"ID":"b801913c-b11c-49f6-9e07-7cf83fb28d7f","Type":"ContainerDied","Data":"7feec3849795c09ab6bd2b6704850e0bf26659fd2cc144d82a663f89a12faa5c"}
Oct 31 01:47:17.161760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.161379    5293 scope.go:122] "RemoveContainer" containerID="66b1e5e165ff79f9c7d5280c352a7d0d241a1c299b943e97fee0455e9c0f027f"
Oct 31 01:47:17.161760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.161308    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"
Oct 31 01:47:17.163086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.163066    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"
Oct 31 01:47:17.163174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.163093    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9" event={"ID":"07c887a3-622c-4808-b16a-c0b5d359f3fd","Type":"ContainerDied","Data":"1b37eff617b390a589df78191a5acc9f2f79ff360b9ce0d6d35c730dc2ae65cd"}
Oct 31 01:47:17.177971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.177948    5293 scope.go:122] "RemoveContainer" containerID="6049b1d4668787da89d71588337e89cd2a1422588007b8a5d63a414503410351"
Oct 31 01:47:17.195822 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.195798    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"]
Oct 31 01:47:17.202011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.201982    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b6b833-58bd44c469-sk2st"]
Oct 31 01:47:17.210407 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.210383    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"]
Oct 31 01:47:17.216064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:17.216024    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6b833-66c66b5779-846g9"]
Oct 31 01:47:18.065069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:18.059610    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"]
Oct 31 01:47:18.071133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:18.066414    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6017d23d-p6x98"]
Oct 31 01:47:18.174310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:18.174278    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c887a3-622c-4808-b16a-c0b5d359f3fd" path="/var/lib/kubelet/pods/07c887a3-622c-4808-b16a-c0b5d359f3fd/volumes"
Oct 31 01:47:18.174725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:18.174711    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b801913c-b11c-49f6-9e07-7cf83fb28d7f" path="/var/lib/kubelet/pods/b801913c-b11c-49f6-9e07-7cf83fb28d7f/volumes"
Oct 31 01:47:18.175116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:18.175104    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f004ee-b685-4130-879e-9a115968d320" path="/var/lib/kubelet/pods/c1f004ee-b685-4130-879e-9a115968d320/volumes"
Oct 31 01:47:18.925758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:18.925720    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"]
Oct 31 01:47:18.942779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:18.942747    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"]
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.957339    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"]
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958417    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958438    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958460    5293 cpu_manager.go:414] "containerMap: removing container" podUID="cc650adc-eaf9-41e1-a79a-0d9e23f09b79" containerName="revoke-keys"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958468    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc650adc-eaf9-41e1-a79a-0d9e23f09b79" containerName="revoke-keys"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958484    5293 cpu_manager.go:414] "containerMap: removing container" podUID="07c887a3-622c-4808-b16a-c0b5d359f3fd" containerName="payload-pre-processing"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958491    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c887a3-622c-4808-b16a-c0b5d359f3fd" containerName="payload-pre-processing"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958508    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958515    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958525    5293 cpu_manager.go:414] "containerMap: removing container" podUID="417ee2e3-c6e4-4ba9-933b-d57ba7457454" containerName="revoke-keys"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958532    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417ee2e3-c6e4-4ba9-933b-d57ba7457454" containerName="revoke-keys"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958561    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b801913c-b11c-49f6-9e07-7cf83fb28d7f" containerName="payload-processing"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958568    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b801913c-b11c-49f6-9e07-7cf83fb28d7f" containerName="payload-processing"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958922    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df3b6eb-f9da-446a-a551-3c56f0a3615c" containerName="istio-proxy"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958939    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc650adc-eaf9-41e1-a79a-0d9e23f09b79" containerName="revoke-keys"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958950    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c887a3-622c-4808-b16a-c0b5d359f3fd" containerName="payload-pre-processing"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958962    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e99c30-f334-44be-b22b-a1e12acd6762" containerName="istio-proxy"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958972    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="417ee2e3-c6e4-4ba9-933b-d57ba7457454" containerName="revoke-keys"
Oct 31 01:47:19.962705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.958984    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b801913c-b11c-49f6-9e07-7cf83fb28d7f" containerName="payload-processing"
Oct 31 01:47:19.968513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.968415    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"]
Oct 31 01:47:19.968650 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.968531    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:19.971926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:19.971895    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-50ac12-openshift-default-dockercfg-hfpv5\""
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.094209    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-certs\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.116220    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-envoy\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.116484    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-data\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.116519    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs6cr\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-kube-api-access-fs6cr\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.116551    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-podinfo\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.116598    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-socket\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.116691    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e96e89-496f-4e62-a7c0-64773daa2b67-istiod-ca-cert\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.116749    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-credential-socket\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.151068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.116820    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-token\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.331619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.331505    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-socket\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.331619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.331615    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e96e89-496f-4e62-a7c0-64773daa2b67-istiod-ca-cert\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.331857 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.331667    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-credential-socket\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.331857 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.331738    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-token\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.331857 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.331831    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-certs\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.332015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.331913    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-envoy\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.332402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.332167    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-data\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.332402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.332211    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs6cr\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-kube-api-access-fs6cr\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.332402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.332246    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-podinfo\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.343064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.341430    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-socket\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.343064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.342141    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e96e89-496f-4e62-a7c0-64773daa2b67-istiod-ca-cert\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.343064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.342527    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-credential-socket\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.346393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.346357    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-data\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.346679 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.346652    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-certs\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.463971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.463908    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-envoy\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.464602 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.464535    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-podinfo\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.468180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.466240    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-token\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.468379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.468357    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs6cr\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-kube-api-access-fs6cr\") pod \"e2e-api-a-50ac12-openshift-default-65d95897c-ljcll\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") " pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.608216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.607355    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:20.983597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.983566    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"]
Oct 31 01:47:20.987742 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:20.987711    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e96e89_496f_4e62_a7c0_64773daa2b67.slice/crio-80755274a612f8025f6e6bd21e572bd5b33facd8a59df358b1a88a071a4b209c WatchSource:0}: Error finding container 80755274a612f8025f6e6bd21e572bd5b33facd8a59df358b1a88a071a4b209c: Status 404 returned error can't find the container with id 80755274a612f8025f6e6bd21e572bd5b33facd8a59df358b1a88a071a4b209c
Oct 31 01:47:20.990016 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.989982    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:20.990121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.990083    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:20.990121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:20.990114    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:21.484554 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:21.484519    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" event={"ID":"40e96e89-496f-4e62-a7c0-64773daa2b67","Type":"ContainerStarted","Data":"fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71"}
Oct 31 01:47:21.484747 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:21.484582    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" event={"ID":"40e96e89-496f-4e62-a7c0-64773daa2b67","Type":"ContainerStarted","Data":"80755274a612f8025f6e6bd21e572bd5b33facd8a59df358b1a88a071a4b209c"}
Oct 31 01:47:21.533069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:21.531676    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" podStartSLOduration=2.5316558970000003 podStartE2EDuration="2.531655897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:21.50557988 +0000 UTC m=+1599.454228274" watchObservedRunningTime="2026-10-31 01:47:21.531655897 +0000 UTC m=+1599.480304289"
Oct 31 01:47:21.621295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:21.620136    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:22.108088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:22.105528    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"]
Oct 31 01:47:22.143071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:22.138950    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6b833-w26g2"]
Oct 31 01:47:22.239124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:22.232624    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59df9bc-dff5-405c-b769-a8d6e18d89d4" path="/var/lib/kubelet/pods/a59df9bc-dff5-405c-b769-a8d6e18d89d4/volumes"
Oct 31 01:47:22.752725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:22.743719    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:22.752725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:22.752059    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:22.756335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:22.755653    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:47:23.939066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:23.938121    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz" podUID="8e81f600-b675-4502-b60b-44b3d82e8b27" containerName="payload-processing" containerID="cri-o://bc67b856ec426647ef166b2d808a70acb21ac1f2a8745040b9c4038e8554dbbf" gracePeriod=25
Oct 31 01:47:23.953112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:23.952324    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n" podUID="2a453023-ea17-4aa3-917e-f404ba3bfabe" containerName="payload-pre-processing" containerID="cri-o://a1b9a8f869f9462e0a91b2b3463b1d3da08c27a2e01ea5451185954d31dbb99a" gracePeriod=25
Oct 31 01:47:25.165258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.163451    5293 generic.go:356] "Generic (PLEG): container finished" podID="8e81f600-b675-4502-b60b-44b3d82e8b27" containerID="bc67b856ec426647ef166b2d808a70acb21ac1f2a8745040b9c4038e8554dbbf" exitCode=0
Oct 31 01:47:25.165258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.163779    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz" event={"ID":"8e81f600-b675-4502-b60b-44b3d82e8b27","Type":"ContainerDied","Data":"bc67b856ec426647ef166b2d808a70acb21ac1f2a8745040b9c4038e8554dbbf"}
Oct 31 01:47:25.165258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.163830    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz" event={"ID":"8e81f600-b675-4502-b60b-44b3d82e8b27","Type":"ContainerDied","Data":"8f4c4fcb1187c04de0ca9a5dd4ed1c0a7d71aa2c6474983f811c73f1d1db9647"}
Oct 31 01:47:25.165258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.163853    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4c4fcb1187c04de0ca9a5dd4ed1c0a7d71aa2c6474983f811c73f1d1db9647"
Oct 31 01:47:25.180564 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.180196    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:47:25.182055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.181301    5293 generic.go:356] "Generic (PLEG): container finished" podID="2a453023-ea17-4aa3-917e-f404ba3bfabe" containerID="a1b9a8f869f9462e0a91b2b3463b1d3da08c27a2e01ea5451185954d31dbb99a" exitCode=0
Oct 31 01:47:25.182055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.181419    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n" event={"ID":"2a453023-ea17-4aa3-917e-f404ba3bfabe","Type":"ContainerDied","Data":"a1b9a8f869f9462e0a91b2b3463b1d3da08c27a2e01ea5451185954d31dbb99a"}
Oct 31 01:47:25.191868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.191211    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:47:25.257878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.242406    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a453023-ea17-4aa3-917e-f404ba3bfabe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a453023-ea17-4aa3-917e-f404ba3bfabe-plugins-config-volume\") pod \"2a453023-ea17-4aa3-917e-f404ba3bfabe\" (UID: \"2a453023-ea17-4aa3-917e-f404ba3bfabe\") "
Oct 31 01:47:25.415066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.400249    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e81f600-b675-4502-b60b-44b3d82e8b27-kube-api-access-p47v7\" (UniqueName: \"kubernetes.io/projected/8e81f600-b675-4502-b60b-44b3d82e8b27-kube-api-access-p47v7\") pod \"8e81f600-b675-4502-b60b-44b3d82e8b27\" (UID: \"8e81f600-b675-4502-b60b-44b3d82e8b27\") "
Oct 31 01:47:25.415066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.400404    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a453023-ea17-4aa3-917e-f404ba3bfabe-kube-api-access-w9nkl\" (UniqueName: \"kubernetes.io/projected/2a453023-ea17-4aa3-917e-f404ba3bfabe-kube-api-access-w9nkl\") pod \"2a453023-ea17-4aa3-917e-f404ba3bfabe\" (UID: \"2a453023-ea17-4aa3-917e-f404ba3bfabe\") "
Oct 31 01:47:25.415066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.400606    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e81f600-b675-4502-b60b-44b3d82e8b27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81f600-b675-4502-b60b-44b3d82e8b27-plugins-config-volume\") pod \"8e81f600-b675-4502-b60b-44b3d82e8b27\" (UID: \"8e81f600-b675-4502-b60b-44b3d82e8b27\") "
Oct 31 01:47:25.415066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.254977    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a453023-ea17-4aa3-917e-f404ba3bfabe-plugins-config-volume" pod "2a453023-ea17-4aa3-917e-f404ba3bfabe" (UID: "2a453023-ea17-4aa3-917e-f404ba3bfabe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:25.533225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.508024    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a453023-ea17-4aa3-917e-f404ba3bfabe-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:25.557076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.554461    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e81f600-b675-4502-b60b-44b3d82e8b27-plugins-config-volume" pod "8e81f600-b675-4502-b60b-44b3d82e8b27" (UID: "8e81f600-b675-4502-b60b-44b3d82e8b27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:25.624083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.601333    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a453023-ea17-4aa3-917e-f404ba3bfabe-kube-api-access-w9nkl" pod "2a453023-ea17-4aa3-917e-f404ba3bfabe" (UID: "2a453023-ea17-4aa3-917e-f404ba3bfabe"). InnerVolumeSpecName "kube-api-access-w9nkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:25.624083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.623574    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9nkl\" (UniqueName: \"kubernetes.io/projected/2a453023-ea17-4aa3-917e-f404ba3bfabe-kube-api-access-w9nkl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:25.624083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.623607    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81f600-b675-4502-b60b-44b3d82e8b27-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:25.652077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.649142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:25.652077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.649555    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:47:25.744167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.706570    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"]
Oct 31 01:47:25.744167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.707598    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8e81f600-b675-4502-b60b-44b3d82e8b27" containerName="payload-processing"
Oct 31 01:47:25.744167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.707621    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81f600-b675-4502-b60b-44b3d82e8b27" containerName="payload-processing"
Oct 31 01:47:25.744167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.707669    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2a453023-ea17-4aa3-917e-f404ba3bfabe" containerName="payload-pre-processing"
Oct 31 01:47:25.744167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.707679    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a453023-ea17-4aa3-917e-f404ba3bfabe" containerName="payload-pre-processing"
Oct 31 01:47:25.744167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.707840    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81f600-b675-4502-b60b-44b3d82e8b27" containerName="payload-processing"
Oct 31 01:47:25.744167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.707859    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a453023-ea17-4aa3-917e-f404ba3bfabe" containerName="payload-pre-processing"
Oct 31 01:47:25.800133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.749481    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81f600-b675-4502-b60b-44b3d82e8b27-kube-api-access-p47v7" pod "8e81f600-b675-4502-b60b-44b3d82e8b27" (UID: "8e81f600-b675-4502-b60b-44b3d82e8b27"). InnerVolumeSpecName "kube-api-access-p47v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:25.800133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.753446    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:25.818160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.818129    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-50ac12\""
Oct 31 01:47:25.841963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.819408    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-50ac12-dockercfg-922mq\""
Oct 31 01:47:25.913079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.899356    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwbv\" (UniqueName: \"kubernetes.io/projected/16679091-1da3-428d-ba93-611ba19cd055-kube-api-access-dhwbv\") pod \"payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc\" (UID: \"16679091-1da3-428d-ba93-611ba19cd055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:25.913477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.913450    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16679091-1da3-428d-ba93-611ba19cd055-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc\" (UID: \"16679091-1da3-428d-ba93-611ba19cd055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:25.913820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.913803    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47v7\" (UniqueName: \"kubernetes.io/projected/8e81f600-b675-4502-b60b-44b3d82e8b27-kube-api-access-p47v7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:25.958900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:25.958871    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"]
Oct 31 01:47:26.026969 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.021008    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16679091-1da3-428d-ba93-611ba19cd055-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc\" (UID: \"16679091-1da3-428d-ba93-611ba19cd055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:26.091117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.037137    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:26.091117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.038409    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16679091-1da3-428d-ba93-611ba19cd055-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc\" (UID: \"16679091-1da3-428d-ba93-611ba19cd055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:26.091117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.038994    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwbv\" (UniqueName: \"kubernetes.io/projected/16679091-1da3-428d-ba93-611ba19cd055-kube-api-access-dhwbv\") pod \"payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc\" (UID: \"16679091-1da3-428d-ba93-611ba19cd055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:26.091117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.040499    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:47:26.112490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.112460    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"]
Oct 31 01:47:26.221380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.221334    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:47:26.312892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.312360    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwbv\" (UniqueName: \"kubernetes.io/projected/16679091-1da3-428d-ba93-611ba19cd055-kube-api-access-dhwbv\") pod \"payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc\" (UID: \"16679091-1da3-428d-ba93-611ba19cd055\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:26.319196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.317255    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"]
Oct 31 01:47:26.319196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.319158    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:26.344879 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.344781    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"
Oct 31 01:47:26.356215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.356169    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"
Oct 31 01:47:26.681268 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.681214    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:26.723062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.714021    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:26.748173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.744584    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:26.748173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.747107    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:26.750060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.749326    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8jq\" (UniqueName: \"kubernetes.io/projected/715e9e29-dbe7-4866-ba65-672cf2a06267-kube-api-access-qp8jq\") pod \"payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5\" (UID: \"715e9e29-dbe7-4866-ba65-672cf2a06267\") " pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:26.750060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.749653    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e9e29-dbe7-4866-ba65-672cf2a06267-plugins-config-volume\") pod \"payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5\" (UID: \"715e9e29-dbe7-4866-ba65-672cf2a06267\") " pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:26.771076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.768155    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:26.858959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.858920    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:26.881348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.881322    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:47:26.887420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.887397    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n" event={"ID":"2a453023-ea17-4aa3-917e-f404ba3bfabe","Type":"ContainerDied","Data":"1ce2ebd34553467f816ea03647a2da3b9a14596a4c06880df3a0d9561888e8aa"}
Oct 31 01:47:26.887721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.887689    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:47:26.889462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.889437    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:47:26.889746 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.889731    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"]
Oct 31 01:47:26.889900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.889886    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-nvjrj"]
Oct 31 01:47:26.890136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.887192    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e9e29-dbe7-4866-ba65-672cf2a06267-plugins-config-volume\") pod \"payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5\" (UID: \"715e9e29-dbe7-4866-ba65-672cf2a06267\") " pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:26.890348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.887843    5293 scope.go:122] "RemoveContainer" containerID="a1b9a8f869f9462e0a91b2b3463b1d3da08c27a2e01ea5451185954d31dbb99a"
Oct 31 01:47:26.890694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.884964    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e9e29-dbe7-4866-ba65-672cf2a06267-plugins-config-volume\") pod \"payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5\" (UID: \"715e9e29-dbe7-4866-ba65-672cf2a06267\") " pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:26.890944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.890919    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8jq\" (UniqueName: \"kubernetes.io/projected/715e9e29-dbe7-4866-ba65-672cf2a06267-kube-api-access-qp8jq\") pod \"payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5\" (UID: \"715e9e29-dbe7-4866-ba65-672cf2a06267\") " pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:26.904678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:26.904653    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:47:27.038517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:27.005396    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"]
Oct 31 01:47:27.068191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:27.068120    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0239db-568d9557f7-kv5lz"]
Oct 31 01:47:27.111337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:27.111300    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8jq\" (UniqueName: \"kubernetes.io/projected/715e9e29-dbe7-4866-ba65-672cf2a06267-kube-api-access-qp8jq\") pod \"payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5\" (UID: \"715e9e29-dbe7-4866-ba65-672cf2a06267\") " pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:27.141071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:27.138966    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:47:27.330337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:27.330092    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:27.815827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:27.810695    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"]
Oct 31 01:47:27.817425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:27.817401    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0239db-86587fc5bc-g5q4n"]
Oct 31 01:47:27.961552 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:27.961478    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16679091_1da3_428d_ba93_611ba19cd055.slice/crio-5ea9b38b078c33a2f778c86729f82fe9ea566d6d28c64fb9e512a6662a01fad9 WatchSource:0}: Error finding container 5ea9b38b078c33a2f778c86729f82fe9ea566d6d28c64fb9e512a6662a01fad9: Status 404 returned error can't find the container with id 5ea9b38b078c33a2f778c86729f82fe9ea566d6d28c64fb9e512a6662a01fad9
Oct 31 01:47:27.996002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:27.995907    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"]
Oct 31 01:47:28.109579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.105278    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"]
Oct 31 01:47:28.109579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.105333    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c8afce-m6mst"]
Oct 31 01:47:28.303865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.300984    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a453023-ea17-4aa3-917e-f404ba3bfabe" path="/var/lib/kubelet/pods/2a453023-ea17-4aa3-917e-f404ba3bfabe/volumes"
Oct 31 01:47:28.303865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.301766    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a33b0f-8152-4a2f-8175-61da9554652c" path="/var/lib/kubelet/pods/63a33b0f-8152-4a2f-8175-61da9554652c/volumes"
Oct 31 01:47:28.303865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.302437    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e81f600-b675-4502-b60b-44b3d82e8b27" path="/var/lib/kubelet/pods/8e81f600-b675-4502-b60b-44b3d82e8b27/volumes"
Oct 31 01:47:28.304991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.304969    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9932f1-b97b-410e-ae06-c2ad32369e80" path="/var/lib/kubelet/pods/fe9932f1-b97b-410e-ae06-c2ad32369e80/volumes"
Oct 31 01:47:28.309595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.309266    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x947t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:47:28.309595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.309315    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:28.556433 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.556402    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"]
Oct 31 01:47:28.556770 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:28.556659    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715e9e29_dbe7_4866_ba65_672cf2a06267.slice/crio-158688e030a00fb724eebe85b44818d33191bae771a4641620c9c3dc39e7a2f2 WatchSource:0}: Error finding container 158688e030a00fb724eebe85b44818d33191bae771a4641620c9c3dc39e7a2f2: Status 404 returned error can't find the container with id 158688e030a00fb724eebe85b44818d33191bae771a4641620c9c3dc39e7a2f2
Oct 31 01:47:28.724522 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.724481    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc" event={"ID":"16679091-1da3-428d-ba93-611ba19cd055","Type":"ContainerStarted","Data":"b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1"}
Oct 31 01:47:28.724673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.724551    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc" event={"ID":"16679091-1da3-428d-ba93-611ba19cd055","Type":"ContainerStarted","Data":"5ea9b38b078c33a2f778c86729f82fe9ea566d6d28c64fb9e512a6662a01fad9"}
Oct 31 01:47:28.724673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.724581    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:28.726653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.726626    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5" event={"ID":"715e9e29-dbe7-4866-ba65-672cf2a06267","Type":"ContainerStarted","Data":"39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3"}
Oct 31 01:47:28.726802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.726675    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5" event={"ID":"715e9e29-dbe7-4866-ba65-672cf2a06267","Type":"ContainerStarted","Data":"158688e030a00fb724eebe85b44818d33191bae771a4641620c9c3dc39e7a2f2"}
Oct 31 01:47:28.726802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.726719    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:28.751280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.744421    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc" podStartSLOduration=3.744404235 podStartE2EDuration="3.744404235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:28.740796848 +0000 UTC m=+1606.689445242" watchObservedRunningTime="2026-10-31 01:47:28.744404235 +0000 UTC m=+1606.693052629"
Oct 31 01:47:28.765334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.765287    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5" podStartSLOduration=3.765272062 podStartE2EDuration="3.765272062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:28.759392006 +0000 UTC m=+1606.708040435" watchObservedRunningTime="2026-10-31 01:47:28.765272062 +0000 UTC m=+1606.713920454"
Oct 31 01:47:28.892117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.890229    5293 patch_prober.go:29] interesting pod/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:47:28.892117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:28.890300    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:29.928011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:29.927968    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"]
Oct 31 01:47:29.928651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:29.928276    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" containerID="cri-o://ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca" gracePeriod=30
Oct 31 01:47:30.613754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:30.613402    5293 patch_prober.go:29] interesting pod/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:47:30.613754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:30.613482    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:47:30.726373 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:30.726294    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"]
Oct 31 01:47:30.767062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:30.752725    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0239db-t7qrt"]
Oct 31 01:47:32.285979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:32.240195    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc650adc-eaf9-41e1-a79a-0d9e23f09b79" path="/var/lib/kubelet/pods/cc650adc-eaf9-41e1-a79a-0d9e23f09b79/volumes"
Oct 31 01:47:33.869786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:33.869752    5293 generic.go:356] "Generic (PLEG): container finished" podID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerID="fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387" exitCode=0
Oct 31 01:47:33.870362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:33.869827    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" event={"ID":"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0","Type":"ContainerDied","Data":"fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387"}
Oct 31 01:47:33.870449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:33.870430    5293 scope.go:122] "RemoveContainer" containerID="fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387"
Oct 31 01:47:33.871879 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:33.871848    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:33.871977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:33.871923    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:33.872015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:33.871975    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:34.033086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:34.029088    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"]
Oct 31 01:47:34.035697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:34.035673    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ee28f9-49gcd"]
Oct 31 01:47:34.230061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:34.229304    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb84b66e-25dd-4bc2-9a95-024ab80352b5" path="/var/lib/kubelet/pods/bb84b66e-25dd-4bc2-9a95-024ab80352b5/volumes"
Oct 31 01:47:34.946158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:34.941799    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" event={"ID":"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0","Type":"ContainerStarted","Data":"e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b"}
Oct 31 01:47:35.287179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.287152    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:47:35.443803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.443772    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-credential-socket\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.443803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.443806    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-data\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.444085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.443837    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-token\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-token\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.444085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.443904    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-kube-api-access-4dzwl\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-kube-api-access-4dzwl\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.444085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.443986    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istiod-ca-cert\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.444085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444029    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-podinfo\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.444310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444135    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-socket\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.444310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444183    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-credential-socket" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:35.444310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444202    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-data" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:35.444310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444259    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-certs\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.444525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444362    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-envoy\") pod \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\" (UID: \"12d50b4a-007a-4c9e-9bc6-3f9abe048781\") "
Oct 31 01:47:35.444525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444429    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-socket" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:35.444525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444437    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istiod-ca-cert" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:35.444688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444576    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-certs" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:35.444950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444929    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.444950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444950    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.445320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444962    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.445320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444974    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.445320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.444985    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.446634 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.446612    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-podinfo" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:35.446892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.446871    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-kube-api-access-4dzwl" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "kube-api-access-4dzwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:35.446976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.446937    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-token" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:35.447104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.447023    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-envoy" pod "12d50b4a-007a-4c9e-9bc6-3f9abe048781" (UID: "12d50b4a-007a-4c9e-9bc6-3f9abe048781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:35.487353 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.487319    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:47:35.487491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.487470    5293 patch_prober.go:29] interesting pod/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:47:35.487555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.487517    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 01:47:35.546138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.546060    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.546138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.546082    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.546138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.546094    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dzwl\" (UniqueName: \"kubernetes.io/projected/12d50b4a-007a-4c9e-9bc6-3f9abe048781-kube-api-access-4dzwl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.546138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.546104    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d50b4a-007a-4c9e-9bc6-3f9abe048781-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.951459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.951425    5293 generic.go:356] "Generic (PLEG): container finished" podID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerID="ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca" exitCode=0
Oct 31 01:47:35.951963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.951514    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"
Oct 31 01:47:35.951963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.951518    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" event={"ID":"12d50b4a-007a-4c9e-9bc6-3f9abe048781","Type":"ContainerDied","Data":"ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca"}
Oct 31 01:47:35.951963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.951606    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc" event={"ID":"12d50b4a-007a-4c9e-9bc6-3f9abe048781","Type":"ContainerDied","Data":"63fa1013ab3bf6415266daa9e06070e83fa2a9dfd90aa57990a102f97da1ee15"}
Oct 31 01:47:35.951963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.951631    5293 scope.go:122] "RemoveContainer" containerID="ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca"
Oct 31 01:47:35.964644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.964627    5293 scope.go:122] "RemoveContainer" containerID="902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1"
Oct 31 01:47:35.975448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.975431    5293 scope.go:122] "RemoveContainer" containerID="ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca"
Oct 31 01:47:35.975637 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.975619    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"]
Oct 31 01:47:35.975743 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:35.975716    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca\": container with ID starting with ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca not found: ID does not exist" containerID="ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca"
Oct 31 01:47:35.975790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.975753    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca"} err="failed to get container status \"ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca\": rpc error: code = NotFound desc = could not find container \"ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca\": container with ID starting with ce61d77caf909a37b9c7441afc8c492855b8a119bf8b3a26aa0d7b8745cbbaca not found: ID does not exist"
Oct 31 01:47:35.975790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.975772    5293 scope.go:122] "RemoveContainer" containerID="902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1"
Oct 31 01:47:35.976172 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:35.976033    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1\": container with ID starting with 902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1 not found: ID does not exist" containerID="902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1"
Oct 31 01:47:35.976279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.976178    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1"} err="failed to get container status \"902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1\": rpc error: code = NotFound desc = could not find container \"902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1\": container with ID starting with 902317d0e34215907586c5ef2028bb41c07fe50ff097a9064ec3a21d016927f1 not found: ID does not exist"
Oct 31 01:47:35.981846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:35.981826    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b6b833-openshift-default-8446fb8fcf-z9fxc"]
Oct 31 01:47:36.175310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:36.175272    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" path="/var/lib/kubelet/pods/12d50b4a-007a-4c9e-9bc6-3f9abe048781/volumes"
Oct 31 01:47:36.488758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:36.488722    5293 patch_prober.go:29] interesting pod/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:47:36.489096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:36.489070    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 01:47:37.490063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:37.489496    5293 patch_prober.go:29] interesting pod/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:47:37.490063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:37.489568    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 01:47:37.692257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:37.691106    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"]
Oct 31 01:47:37.692257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:37.691338    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy" containerID="cri-o://e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b" gracePeriod=30
Oct 31 01:47:38.795011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.794978    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"]
Oct 31 01:47:38.795612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.795598    5293 cpu_manager.go:414] "containerMap: removing container" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy"
Oct 31 01:47:38.795658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.795613    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy"
Oct 31 01:47:38.795765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.795752    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy"
Oct 31 01:47:38.834575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.834539    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:38.841256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.841226    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"]
Oct 31 01:47:38.844249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.844188    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb5756cb-dockercfg-zx5rb\""
Oct 31 01:47:38.844444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.844214    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb5756cb\""
Oct 31 01:47:38.850244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.850214    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"]
Oct 31 01:47:38.851895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.851863    5293 cpu_manager.go:414] "containerMap: removing container" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy"
Oct 31 01:47:38.851895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.851888    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy"
Oct 31 01:47:38.852383 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.852354    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d50b4a-007a-4c9e-9bc6-3f9abe048781" containerName="istio-proxy"
Oct 31 01:47:38.858835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.858810    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:38.860617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.860300    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tpsf\" (UniqueName: \"kubernetes.io/projected/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-kube-api-access-2tpsf\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4\" (UID: \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:38.860741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:38.860636    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4\" (UID: \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:39.127343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.114442    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4\" (UID: \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:39.127343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.114619    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tpsf\" (UniqueName: \"kubernetes.io/projected/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-kube-api-access-2tpsf\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4\" (UID: \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:39.139082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.133628    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4\" (UID: \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:39.285096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.217201    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zw9\" (UniqueName: \"kubernetes.io/projected/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-kube-api-access-x5zw9\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db\" (UID: \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:39.285096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.217272    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db\" (UID: \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:39.361124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.334683    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zw9\" (UniqueName: \"kubernetes.io/projected/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-kube-api-access-x5zw9\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db\" (UID: \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:39.444083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.399260    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"]
Oct 31 01:47:39.444083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.442232    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db\" (UID: \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:39.444083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.430031    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tpsf\" (UniqueName: \"kubernetes.io/projected/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-kube-api-access-2tpsf\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4\" (UID: \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:39.444083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.443296    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db\" (UID: \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:39.466091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.460963    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zw9\" (UniqueName: \"kubernetes.io/projected/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-kube-api-access-x5zw9\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db\" (UID: \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:39.493002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.492974    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:39.658686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.658139    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:39.790094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.789917    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:47:39.790550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.790327    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:47:39.841425 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:39.841390    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fc28ff_b89f_4c52_95e8_80746d2d7f9f.slice/crio-1f34dbeb80beab7c35c37504c4704653d2493c6785bd45ec65e87a34d898ea80 WatchSource:0}: Error finding container 1f34dbeb80beab7c35c37504c4704653d2493c6785bd45ec65e87a34d898ea80: Status 404 returned error can't find the container with id 1f34dbeb80beab7c35c37504c4704653d2493c6785bd45ec65e87a34d898ea80
Oct 31 01:47:39.856322 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:39.853528    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"]
Oct 31 01:47:40.081512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.073426    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"]
Oct 31 01:47:40.105660 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:40.087340    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55aa2c7b_086c_4cae_a9f2_76b1293b9eb2.slice/crio-787220a4eb9a512e4b52b62e4dde07ceda6abb54df052e115643a374bf88aaff WatchSource:0}: Error finding container 787220a4eb9a512e4b52b62e4dde07ceda6abb54df052e115643a374bf88aaff: Status 404 returned error can't find the container with id 787220a4eb9a512e4b52b62e4dde07ceda6abb54df052e115643a374bf88aaff
Oct 31 01:47:40.448677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.448638    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db" event={"ID":"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2","Type":"ContainerStarted","Data":"3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d"}
Oct 31 01:47:40.448899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.448712    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:40.448899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.448735    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db" event={"ID":"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2","Type":"ContainerStarted","Data":"787220a4eb9a512e4b52b62e4dde07ceda6abb54df052e115643a374bf88aaff"}
Oct 31 01:47:40.450620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.450588    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4" event={"ID":"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f","Type":"ContainerStarted","Data":"f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5"}
Oct 31 01:47:40.450834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.450809    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:40.450953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.450843    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4" event={"ID":"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f","Type":"ContainerStarted","Data":"1f34dbeb80beab7c35c37504c4704653d2493c6785bd45ec65e87a34d898ea80"}
Oct 31 01:47:40.469154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.469099    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db" podStartSLOduration=2.469032322 podStartE2EDuration="2.469032322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:40.465658119 +0000 UTC m=+1618.414306514" watchObservedRunningTime="2026-10-31 01:47:40.469032322 +0000 UTC m=+1618.417680715"
Oct 31 01:47:40.489895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:40.489836    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4" podStartSLOduration=2.489817805 podStartE2EDuration="2.489817805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:40.484672054 +0000 UTC m=+1618.433320445" watchObservedRunningTime="2026-10-31 01:47:40.489817805 +0000 UTC m=+1618.438466199"
Oct 31 01:47:41.626387 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.626347    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"]
Oct 31 01:47:41.632108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.632060    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.635418 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.635391    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:47:41.635686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.635665    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:47:41.635754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.635705    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:47:41.635754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.635732    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:47:41.641783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.641754    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"]
Oct 31 01:47:41.707082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.705169    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4wh\" (UniqueName: \"kubernetes.io/projected/3cb13ac4-8830-478d-a100-b4a432b0c702-kube-api-access-qn4wh\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.707082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.705230    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.707082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.705354    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.860067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.841501    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4wh\" (UniqueName: \"kubernetes.io/projected/3cb13ac4-8830-478d-a100-b4a432b0c702-kube-api-access-qn4wh\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.860067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.841559    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.860067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.841670    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.889509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.883524    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.889509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.887486    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.932960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.932893    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4wh\" (UniqueName: \"kubernetes.io/projected/3cb13ac4-8830-478d-a100-b4a432b0c702-kube-api-access-qn4wh\") pod \"maas-api-revoke-sub-aaa1d0d6-glq9v\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.952912 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:41.952660    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:41.972490 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:41.972458    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:47:42.317544 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:42.317513    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb13ac4_8830_478d_a100_b4a432b0c702.slice/crio-0479509c433fc3f8c4e3d630486e42aa07cfdf038bc33858ebb54803ca7df7f8 WatchSource:0}: Error finding container 0479509c433fc3f8c4e3d630486e42aa07cfdf038bc33858ebb54803ca7df7f8: Status 404 returned error can't find the container with id 0479509c433fc3f8c4e3d630486e42aa07cfdf038bc33858ebb54803ca7df7f8
Oct 31 01:47:42.317699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:42.317637    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"]
Oct 31 01:47:42.467983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:42.467952    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v" event={"ID":"3cb13ac4-8830-478d-a100-b4a432b0c702","Type":"ContainerStarted","Data":"0479509c433fc3f8c4e3d630486e42aa07cfdf038bc33858ebb54803ca7df7f8"}
Oct 31 01:47:42.847950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:42.847929    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:47:43.479623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:43.479587    5293 generic.go:356] "Generic (PLEG): container finished" podID="3cb13ac4-8830-478d-a100-b4a432b0c702" containerID="1f9bddbb6746b92cb660aa2cff58923564a11f53bf83abe3567e67b4cb02578e" exitCode=0
Oct 31 01:47:43.479804 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:43.479658    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v" event={"ID":"3cb13ac4-8830-478d-a100-b4a432b0c702","Type":"ContainerDied","Data":"1f9bddbb6746b92cb660aa2cff58923564a11f53bf83abe3567e67b4cb02578e"}
Oct 31 01:47:44.663469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.663441    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:44.812389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.812353    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb13ac4-8830-478d-a100-b4a432b0c702-kube-api-access-qn4wh\" (UniqueName: \"kubernetes.io/projected/3cb13ac4-8830-478d-a100-b4a432b0c702-kube-api-access-qn4wh\") pod \"3cb13ac4-8830-478d-a100-b4a432b0c702\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") "
Oct 31 01:47:44.812624 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.812606    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-xks-ca\") pod \"3cb13ac4-8830-478d-a100-b4a432b0c702\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") "
Oct 31 01:47:44.812701 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.812687    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-openshift-ca\") pod \"3cb13ac4-8830-478d-a100-b4a432b0c702\" (UID: \"3cb13ac4-8830-478d-a100-b4a432b0c702\") "
Oct 31 01:47:44.813196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.813162    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-openshift-ca" pod "3cb13ac4-8830-478d-a100-b4a432b0c702" (UID: "3cb13ac4-8830-478d-a100-b4a432b0c702"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:44.813705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.813682    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:44.814775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.814743    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb13ac4-8830-478d-a100-b4a432b0c702-kube-api-access-qn4wh" pod "3cb13ac4-8830-478d-a100-b4a432b0c702" (UID: "3cb13ac4-8830-478d-a100-b4a432b0c702"). InnerVolumeSpecName "kube-api-access-qn4wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:44.814955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.814933    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-xks-ca" pod "3cb13ac4-8830-478d-a100-b4a432b0c702" (UID: "3cb13ac4-8830-478d-a100-b4a432b0c702"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:47:44.915752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.915716    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb13ac4-8830-478d-a100-b4a432b0c702-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:44.915752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:44.915753    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4wh\" (UniqueName: \"kubernetes.io/projected/3cb13ac4-8830-478d-a100-b4a432b0c702-kube-api-access-qn4wh\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:45.239355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.239325    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"]
Oct 31 01:47:45.240279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.240254    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3cb13ac4-8830-478d-a100-b4a432b0c702" containerName="revoke-keys"
Oct 31 01:47:45.240279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.240278    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb13ac4-8830-478d-a100-b4a432b0c702" containerName="revoke-keys"
Oct 31 01:47:45.240540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.240479    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb13ac4-8830-478d-a100-b4a432b0c702" containerName="revoke-keys"
Oct 31 01:47:45.246094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.244975    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.259273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.258698    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"]
Oct 31 01:47:45.260158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.260135    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ea6cc6-openshift-default-dockercfg-dk7n6\""
Oct 31 01:47:45.338083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.337342    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-credential-socket\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.338083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.337439    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61337c-517a-4100-894d-d0169a6e69a1-istio-podinfo\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.338083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.337520    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfg9\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-kube-api-access-2mfg9\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.338083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.337570    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-data\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.338083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.337736    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-envoy\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.397312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.383414    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-socket\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.397312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.383500    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-certs\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.397312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.383577    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-istio-token\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.397312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.383638    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61337c-517a-4100-894d-d0169a6e69a1-istiod-ca-cert\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.485757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.485721    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-credential-socket\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.486524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.486501    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61337c-517a-4100-894d-d0169a6e69a1-istio-podinfo\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.487189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.487166    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfg9\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-kube-api-access-2mfg9\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.528138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.519690    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-data\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.528428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.486417    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-credential-socket\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.528776 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.528756    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-envoy\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.529366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.529347    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-socket\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.529968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.529831    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-socket\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.530119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.530104    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-certs\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.530605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.530497    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-certs\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.531052 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.530750    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-istio-token\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.535714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.535654    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-data\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.536742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.536703    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61337c-517a-4100-894d-d0169a6e69a1-istiod-ca-cert\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.553352 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.553323    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61337c-517a-4100-894d-d0169a6e69a1-istiod-ca-cert\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.553748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.553568    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v" event={"ID":"3cb13ac4-8830-478d-a100-b4a432b0c702","Type":"ContainerDied","Data":"0479509c433fc3f8c4e3d630486e42aa07cfdf038bc33858ebb54803ca7df7f8"}
Oct 31 01:47:45.553862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.553785    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0479509c433fc3f8c4e3d630486e42aa07cfdf038bc33858ebb54803ca7df7f8"
Oct 31 01:47:45.553862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.553608    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"
Oct 31 01:47:45.586260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.585260    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfg9\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-kube-api-access-2mfg9\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.598363 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.593463    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61337c-517a-4100-894d-d0169a6e69a1-istio-podinfo\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.624341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.619935    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-istio-token\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.627816 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.627746    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-envoy\") pod \"e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") " pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:45.869009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:45.868966    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:46.104685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.104649    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:47:46.152072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.151517    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-kube-api-access-qrpkh\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-kube-api-access-qrpkh\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.152072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.151645    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-podinfo\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.152072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.151676    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-credential-socket\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.152072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.151758    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-certs\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.152072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.151811    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-data\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.152072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.151848    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-envoy\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.152072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.151908    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-socket\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.153123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.152848    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-credential-socket" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:46.153845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.153739    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-socket" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:46.153845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.153759    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-certs" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:46.153845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.153810    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istiod-ca-cert\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.154312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.154236    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-token\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-token\") pod \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\" (UID: \"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0\") "
Oct 31 01:47:46.155400 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.155368    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.156199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.156184    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.156325 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.156312    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.156417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.153971    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-data" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:46.156508 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.155752    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istiod-ca-cert" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:46.156593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.156139    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-kube-api-access-qrpkh" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "kube-api-access-qrpkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:46.158460 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.158433    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-envoy" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:46.161692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.161652    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-podinfo" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:46.162148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.162113    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-token" pod "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" (UID: "cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:46.268811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.268060    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.268811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.268096    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.268811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.268112    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.268811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.268127    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.268811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.268142    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.268811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.268156    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrpkh\" (UniqueName: \"kubernetes.io/projected/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0-kube-api-access-qrpkh\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:46.370708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.370676    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"]
Oct 31 01:47:46.371271 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:46.371240    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad61337c_517a_4100_894d_d0169a6e69a1.slice/crio-f84792b52af38c2f2da2709496fcab10674648fa0c55a05c5dc5382c67ca4e1f WatchSource:0}: Error finding container f84792b52af38c2f2da2709496fcab10674648fa0c55a05c5dc5382c67ca4e1f: Status 404 returned error can't find the container with id f84792b52af38c2f2da2709496fcab10674648fa0c55a05c5dc5382c67ca4e1f
Oct 31 01:47:46.373858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.373830    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:46.373943 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.373896    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:46.373943 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.373925    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:46.564100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.564060    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" event={"ID":"ad61337c-517a-4100-894d-d0169a6e69a1","Type":"ContainerStarted","Data":"8cd4cdb95c2c7f30dd330fae4f2860e9052663fc3de469a7f54ddc7e6acf4e63"}
Oct 31 01:47:46.564280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.564126    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" event={"ID":"ad61337c-517a-4100-894d-d0169a6e69a1","Type":"ContainerStarted","Data":"f84792b52af38c2f2da2709496fcab10674648fa0c55a05c5dc5382c67ca4e1f"}
Oct 31 01:47:46.566064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.566020    5293 generic.go:356] "Generic (PLEG): container finished" podID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerID="e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b" exitCode=0
Oct 31 01:47:46.566184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.566081    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"
Oct 31 01:47:46.566184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.566095    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" event={"ID":"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0","Type":"ContainerDied","Data":"e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b"}
Oct 31 01:47:46.566184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.566143    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994" event={"ID":"cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0","Type":"ContainerDied","Data":"0123f8cca44516d7e2b0d368d63b0aa6d66530d8a42f6096aaccfa0b5b490ecc"}
Oct 31 01:47:46.566184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.566165    5293 scope.go:122] "RemoveContainer" containerID="e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b"
Oct 31 01:47:46.580155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.580121    5293 scope.go:122] "RemoveContainer" containerID="fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387"
Oct 31 01:47:46.590129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.590075    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" podStartSLOduration=1.590061379 podStartE2EDuration="1.590061379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:46.586956054 +0000 UTC m=+1624.535604446" watchObservedRunningTime="2026-10-31 01:47:46.590061379 +0000 UTC m=+1624.538709770"
Oct 31 01:47:46.597008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.596983    5293 scope.go:122] "RemoveContainer" containerID="e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b"
Oct 31 01:47:46.597407 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:46.597377    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b\": container with ID starting with e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b not found: ID does not exist" containerID="e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b"
Oct 31 01:47:46.597509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.597420    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b"} err="failed to get container status \"e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b\": rpc error: code = NotFound desc = could not find container \"e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b\": container with ID starting with e47c9173f7c37130e0ce87fee9f6daf4529c43acda1ef26fdead0c4fdaee156b not found: ID does not exist"
Oct 31 01:47:46.597509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.597445    5293 scope.go:122] "RemoveContainer" containerID="fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387"
Oct 31 01:47:46.597743 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:47:46.597723    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387\": container with ID starting with fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387 not found: ID does not exist" containerID="fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387"
Oct 31 01:47:46.597797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.597755    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387"} err="failed to get container status \"fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387\": rpc error: code = NotFound desc = could not find container \"fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387\": container with ID starting with fd1b1ac3146005c1f0fe694d4e456158fd242748830a4df79257f4ae02e8b387 not found: ID does not exist"
Oct 31 01:47:46.607511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.607479    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"]
Oct 31 01:47:46.614084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.614057    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0239db-openshift-default-5cb4966c67-nq994"]
Oct 31 01:47:46.869488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:46.869456    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:47.875972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:47.875946    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:48.014909 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:48.014878    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:47:48.201092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:48.200334    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" path="/var/lib/kubelet/pods/cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0/volumes"
Oct 31 01:47:48.583467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:48.583389    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:48.584471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:48.584451    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:47:50.471073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.470323    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"]
Oct 31 01:47:50.471073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.470997    5293 cpu_manager.go:414] "containerMap: removing container" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy"
Oct 31 01:47:50.471073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.471012    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy"
Oct 31 01:47:50.471724 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.471256    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy"
Oct 31 01:47:50.471724 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.471273    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy"
Oct 31 01:47:50.510063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.486078    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:50.602060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.595471    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ea6cc6-dockercfg-b8w6b\""
Oct 31 01:47:50.602060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.595564    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ea6cc6\""
Oct 31 01:47:50.602060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.597086    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"]
Oct 31 01:47:50.602060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.600487    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv\" (UID: \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:50.602060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.600533    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwskn\" (UniqueName: \"kubernetes.io/projected/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-kube-api-access-gwskn\") pod \"payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv\" (UID: \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:50.735114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.726876    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv\" (UID: \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:50.735114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.726928    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwskn\" (UniqueName: \"kubernetes.io/projected/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-kube-api-access-gwskn\") pod \"payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv\" (UID: \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:50.760957 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.760101    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv\" (UID: \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:50.892177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.869848    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwskn\" (UniqueName: \"kubernetes.io/projected/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-kube-api-access-gwskn\") pod \"payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv\" (UID: \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:50.905247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.905214    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"]
Oct 31 01:47:50.923865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.921208    5293 cpu_manager.go:414] "containerMap: removing container" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy"
Oct 31 01:47:50.924724 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.924703    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb89a8ad-f1ab-4f91-a4e2-afd6e24b6fc0" containerName="istio-proxy"
Oct 31 01:47:50.931449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.931317    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"]
Oct 31 01:47:50.931626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:50.931416    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:51.015678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.012856    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686rg\" (UniqueName: \"kubernetes.io/projected/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-kube-api-access-686rg\") pod \"payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg\" (UID: \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:51.015678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.013255    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg\" (UID: \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:51.053071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.042559    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:51.133060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.117023    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg\" (UID: \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:51.133060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.117140    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686rg\" (UniqueName: \"kubernetes.io/projected/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-kube-api-access-686rg\") pod \"payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg\" (UID: \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:51.133060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.118852    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg\" (UID: \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:51.208076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.196763    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686rg\" (UniqueName: \"kubernetes.io/projected/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-kube-api-access-686rg\") pod \"payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg\" (UID: \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:51.397432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.396345    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:51.496391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.496213    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:47:51.526064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:51.524212    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:47:52.117084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:52.100295    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.202108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:52.199149    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.350203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:52.350140    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.499063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:52.461451    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.785888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:52.785775    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"]
Oct 31 01:47:52.884185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:52.884142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.917074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:52.911810    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.917074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:52.911961    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.993068 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:52.987158    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25d9e39_2355_4ca5_80cb_dbeebf95fc26.slice/crio-4d507ef0e3c44c5c3ab6efcb7aae6cbefabd024d718fd4b5301e34e5d51c948f WatchSource:0}: Error finding container 4d507ef0e3c44c5c3ab6efcb7aae6cbefabd024d718fd4b5301e34e5d51c948f: Status 404 returned error can't find the container with id 4d507ef0e3c44c5c3ab6efcb7aae6cbefabd024d718fd4b5301e34e5d51c948f
Oct 31 01:47:53.038147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:53.031218    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:53.206067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:53.199329    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:47:53.239364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:53.236160    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:47:53.414684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:53.403488    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv" event={"ID":"f25d9e39-2355-4ca5-80cb-dbeebf95fc26","Type":"ContainerStarted","Data":"4d507ef0e3c44c5c3ab6efcb7aae6cbefabd024d718fd4b5301e34e5d51c948f"}
Oct 31 01:47:53.506180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:53.506149    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:47:53.959761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:53.934403    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:54.167640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:54.159592    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"]
Oct 31 01:47:54.438746 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:54.438692    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv" event={"ID":"f25d9e39-2355-4ca5-80cb-dbeebf95fc26","Type":"ContainerStarted","Data":"dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479"}
Oct 31 01:47:54.440289 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:54.440229    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:47:54.486328 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:54.481085    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg" event={"ID":"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3","Type":"ContainerStarted","Data":"3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636"}
Oct 31 01:47:54.487138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:54.486718    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:47:54.488474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:54.488378    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg" event={"ID":"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3","Type":"ContainerStarted","Data":"3397ef1e66131e919ae2c40b38c39345f99ff2109abbd79ffacb84f54c803dff"}
Oct 31 01:47:54.523085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:54.522287    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv" podStartSLOduration=4.522265513 podStartE2EDuration="4.522265513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:54.486381719 +0000 UTC m=+1632.435030132" watchObservedRunningTime="2026-10-31 01:47:54.522265513 +0000 UTC m=+1632.470913905"
Oct 31 01:47:54.565217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:54.560240    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg" podStartSLOduration=4.560220673 podStartE2EDuration="4.560220673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:54.520696697 +0000 UTC m=+1632.469345090" watchObservedRunningTime="2026-10-31 01:47:54.560220673 +0000 UTC m=+1632.508869070"
Oct 31 01:47:56.185798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.185752    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"]
Oct 31 01:47:56.190591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.190563    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"]
Oct 31 01:47:56.190749 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.190674    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.194330 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.194299    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9f0694-openshift-default-dockercfg-dzx56\""
Oct 31 01:47:56.342585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.342544    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-envoy\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.342839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.342818    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-podinfo\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.343106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.343088    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/546bcf87-5caf-4136-8a1f-d6719ba577cb-istiod-ca-cert\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.343353 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.343336    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-socket\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.343535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.343517    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-credential-socket\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.343667 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.343653    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsc4p\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-kube-api-access-wsc4p\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.343802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.343790    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-token\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.344073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.344055    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-data\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.344208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.344194    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-certs\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.445998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.445920    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/546bcf87-5caf-4136-8a1f-d6719ba577cb-istiod-ca-cert\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.446364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.446337    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-socket\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.446613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.446594    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-credential-socket\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.446890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.446840    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-credential-socket\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.446890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.446690    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-socket\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.447011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.446965    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/546bcf87-5caf-4136-8a1f-d6719ba577cb-istiod-ca-cert\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.447121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.447108    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsc4p\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-kube-api-access-wsc4p\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.447285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.447271    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-token\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.447649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.447634    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-data\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.447991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.447979    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-certs\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.448633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.447941    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-data\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.448633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.448123    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-envoy\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.448633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.448173    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-podinfo\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.448633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.448611    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-certs\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.450786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.450764    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-podinfo\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.451256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.451235    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-envoy\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.462349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.462326    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-token\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.463525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.463490    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsc4p\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-kube-api-access-wsc4p\") pod \"e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") " pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:56.563064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:56.550352    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:57.185018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:57.184988    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"]
Oct 31 01:47:57.185410 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:47:57.185383    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546bcf87_5caf_4136_8a1f_d6719ba577cb.slice/crio-495a02e24249196511ee6c34275df095247a940744cc52a421d5db630621d33b WatchSource:0}: Error finding container 495a02e24249196511ee6c34275df095247a940744cc52a421d5db630621d33b: Status 404 returned error can't find the container with id 495a02e24249196511ee6c34275df095247a940744cc52a421d5db630621d33b
Oct 31 01:47:57.188163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:57.188128    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:57.188525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:57.188199    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:57.188525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:57.188434    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:47:57.644145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:57.644020    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" event={"ID":"546bcf87-5caf-4136-8a1f-d6719ba577cb","Type":"ContainerStarted","Data":"2e1cbff65f4405c2a8cecd8ac5bb497a488abb505635869b31a7335ac22bf817"}
Oct 31 01:47:57.644145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:57.644104    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" event={"ID":"546bcf87-5caf-4136-8a1f-d6719ba577cb","Type":"ContainerStarted","Data":"495a02e24249196511ee6c34275df095247a940744cc52a421d5db630621d33b"}
Oct 31 01:47:57.781080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:57.778522    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podStartSLOduration=1.77849951 podStartE2EDuration="1.77849951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:57.738419626 +0000 UTC m=+1635.687068039" watchObservedRunningTime="2026-10-31 01:47:57.77849951 +0000 UTC m=+1635.727147904"
Oct 31 01:47:58.557343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:58.557180    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:58.615800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:58.615450    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:58.681054 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:58.681011    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:47:58.681940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:47:58.681917    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:48:01.202082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.202030    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"]
Oct 31 01:48:01.207395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.207368    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:01.214269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.214026    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-03004a\""
Oct 31 01:48:01.218957 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.218675    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"]
Oct 31 01:48:01.223113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.223093    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-03004a-dockercfg-7djn4\""
Oct 31 01:48:01.257433 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.257405    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"]
Oct 31 01:48:01.262216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.262192    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:01.280465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.280435    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"]
Oct 31 01:48:01.305092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.299374    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a453e-4093-410d-9757-c2e3272dab99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t\" (UID: \"c02a453e-4093-410d-9757-c2e3272dab99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:01.305092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.299467    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwq7x\" (UniqueName: \"kubernetes.io/projected/c02a453e-4093-410d-9757-c2e3272dab99-kube-api-access-jwq7x\") pod \"payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t\" (UID: \"c02a453e-4093-410d-9757-c2e3272dab99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:01.305092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.299563    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f65019d-5f02-4a97-9a6e-39e4703f588c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d\" (UID: \"5f65019d-5f02-4a97-9a6e-39e4703f588c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:01.305092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.299614    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ggg\" (UniqueName: \"kubernetes.io/projected/5f65019d-5f02-4a97-9a6e-39e4703f588c-kube-api-access-x5ggg\") pod \"payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d\" (UID: \"5f65019d-5f02-4a97-9a6e-39e4703f588c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:01.423121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.415811    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ggg\" (UniqueName: \"kubernetes.io/projected/5f65019d-5f02-4a97-9a6e-39e4703f588c-kube-api-access-x5ggg\") pod \"payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d\" (UID: \"5f65019d-5f02-4a97-9a6e-39e4703f588c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:01.423121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.415973    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a453e-4093-410d-9757-c2e3272dab99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t\" (UID: \"c02a453e-4093-410d-9757-c2e3272dab99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:01.423121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.416084    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwq7x\" (UniqueName: \"kubernetes.io/projected/c02a453e-4093-410d-9757-c2e3272dab99-kube-api-access-jwq7x\") pod \"payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t\" (UID: \"c02a453e-4093-410d-9757-c2e3272dab99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:01.423121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.416192    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f65019d-5f02-4a97-9a6e-39e4703f588c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d\" (UID: \"5f65019d-5f02-4a97-9a6e-39e4703f588c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:01.423121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.418901    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a453e-4093-410d-9757-c2e3272dab99-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t\" (UID: \"c02a453e-4093-410d-9757-c2e3272dab99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:01.505151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.504222    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f65019d-5f02-4a97-9a6e-39e4703f588c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d\" (UID: \"5f65019d-5f02-4a97-9a6e-39e4703f588c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:01.586183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.582759    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwq7x\" (UniqueName: \"kubernetes.io/projected/c02a453e-4093-410d-9757-c2e3272dab99-kube-api-access-jwq7x\") pod \"payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t\" (UID: \"c02a453e-4093-410d-9757-c2e3272dab99\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:01.586183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.583392    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ggg\" (UniqueName: \"kubernetes.io/projected/5f65019d-5f02-4a97-9a6e-39e4703f588c-kube-api-access-x5ggg\") pod \"payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d\" (UID: \"5f65019d-5f02-4a97-9a6e-39e4703f588c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:01.592985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.592750    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:01.834417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.834335    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:01.987189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:01.985400    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"]
Oct 31 01:48:02.396705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.396677    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"]
Oct 31 01:48:02.721097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.721062    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d" event={"ID":"5f65019d-5f02-4a97-9a6e-39e4703f588c","Type":"ContainerStarted","Data":"c3caf70b4b75ae58f20c22551a96eeebb1cc296607b91561bea0c5ce2c47ef4c"}
Oct 31 01:48:02.721279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.721117    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d" event={"ID":"5f65019d-5f02-4a97-9a6e-39e4703f588c","Type":"ContainerStarted","Data":"9e54a8d505eefda621bdbfd8b2029eb16e866078891e8a356283dd897cad5c01"}
Oct 31 01:48:02.721373 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.721355    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:02.723197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.723172    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t" event={"ID":"c02a453e-4093-410d-9757-c2e3272dab99","Type":"ContainerStarted","Data":"5abfdad8314d2fe4b55d3fd4759d161d68d7fbc4901518153ea82780d0007bea"}
Oct 31 01:48:02.723326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.723215    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t" event={"ID":"c02a453e-4093-410d-9757-c2e3272dab99","Type":"ContainerStarted","Data":"af710856ff0f1c0be8481b67114f5bff985cc29871a70ab572234fb0ce957a49"}
Oct 31 01:48:02.723326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.723290    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:02.743277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.743231    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d" podStartSLOduration=1.743217933 podStartE2EDuration="1.743217933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:02.738638575 +0000 UTC m=+1640.687286973" watchObservedRunningTime="2026-10-31 01:48:02.743217933 +0000 UTC m=+1640.691866324"
Oct 31 01:48:02.758658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:02.758602    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t" podStartSLOduration=1.7585881730000001 podStartE2EDuration="1.758588173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:02.756254443 +0000 UTC m=+1640.704902845" watchObservedRunningTime="2026-10-31 01:48:02.758588173 +0000 UTC m=+1640.707236586"
Oct 31 01:48:03.780338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:03.780253    5293 generic.go:356] "Generic (PLEG): container finished" podID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerID="fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71" exitCode=0
Oct 31 01:48:03.788784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:03.786692    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" event={"ID":"40e96e89-496f-4e62-a7c0-64773daa2b67","Type":"ContainerDied","Data":"fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71"}
Oct 31 01:48:03.788784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:03.787385    5293 scope.go:122] "RemoveContainer" containerID="fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71"
Oct 31 01:48:03.791004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:03.790971    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:03.791185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:03.791165    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:03.791261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:03.791214    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:04.804067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:04.800964    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" event={"ID":"40e96e89-496f-4e62-a7c0-64773daa2b67","Type":"ContainerStarted","Data":"9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29"}
Oct 31 01:48:05.619073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:05.618751    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:48:05.619073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:05.618914    5293 patch_prober.go:29] interesting pod/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:48:05.619073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:05.618957    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:48:06.629213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:06.610676    5293 patch_prober.go:29] interesting pod/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:48:06.629213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:06.610741    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:48:06.635313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:06.635283    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:48:06.658343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:06.658314    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:48:07.609089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:07.609034    5293 patch_prober.go:29] interesting pod/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:48:07.609380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:07.609352    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:48:07.754569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:07.754534    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:48:08.614063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:08.613764    5293 patch_prober.go:29] interesting pod/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:48:08.614063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:08.613836    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:48:09.552737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.548119    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"]
Oct 31 01:48:09.554665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.554636    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"]
Oct 31 01:48:09.554782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.554745    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.558424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.558406    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:48:09.559075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.558930    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:48:09.559230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.558991    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:48:09.559334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.559023    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:48:09.615316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.615285    5293 patch_prober.go:29] interesting pod/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:48:09.615590 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.615563    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:48:09.676070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.666551    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.676488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.676456    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.676681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.676661    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89x8c\" (UniqueName: \"kubernetes.io/projected/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-kube-api-access-89x8c\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.777800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.777755    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.778008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.777877    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.778008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.778001    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89x8c\" (UniqueName: \"kubernetes.io/projected/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-kube-api-access-89x8c\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.778519 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.778490    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.780942 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.780918    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.790077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.790011    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89x8c\" (UniqueName: \"kubernetes.io/projected/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-kube-api-access-89x8c\") pod \"maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.885554 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.885483    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:09.946079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.946031    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"]
Oct 31 01:48:09.954070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.953289    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:09.955596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:09.955496    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"]
Oct 31 01:48:10.088581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.088540    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhkx\" (UniqueName: \"kubernetes.io/projected/b079fa40-b11a-409f-a006-b8e7454e1b89-kube-api-access-kkhkx\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.088748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.088727    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.088814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.088787    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.190059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.190000    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhkx\" (UniqueName: \"kubernetes.io/projected/b079fa40-b11a-409f-a006-b8e7454e1b89-kube-api-access-kkhkx\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.190284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.190165    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.190284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.190203    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.190927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.190903    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.192749 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.192726    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.201816 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.201788    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhkx\" (UniqueName: \"kubernetes.io/projected/b079fa40-b11a-409f-a006-b8e7454e1b89-kube-api-access-kkhkx\") pod \"maas-api-revoke-keys-e2e-api-a-50ac12-f2lps\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.272344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.272309    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"]
Oct 31 01:48:10.272528 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:48:10.272500    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa1c93e_2e8a_40bc_b5cb_f7e8236c5fcb.slice/crio-df6a3c43f171f4bb3b4dd43b4983327f5530eae710ae4a46869f224d1a1fdd73 WatchSource:0}: Error finding container df6a3c43f171f4bb3b4dd43b4983327f5530eae710ae4a46869f224d1a1fdd73: Status 404 returned error can't find the container with id df6a3c43f171f4bb3b4dd43b4983327f5530eae710ae4a46869f224d1a1fdd73
Oct 31 01:48:10.366581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.366277    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:10.613150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.613122    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:48:10.614423 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.614403    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:48:10.733184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:10.733155    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"]
Oct 31 01:48:10.733706 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:48:10.733679    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb079fa40_b11a_409f_a006_b8e7454e1b89.slice/crio-f1a58918df29c5b0339ea093de4b6e280ec4b17a7cdf3924fa3ad562983cf31f WatchSource:0}: Error finding container f1a58918df29c5b0339ea093de4b6e280ec4b17a7cdf3924fa3ad562983cf31f: Status 404 returned error can't find the container with id f1a58918df29c5b0339ea093de4b6e280ec4b17a7cdf3924fa3ad562983cf31f
Oct 31 01:48:11.061252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:11.061219    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps" event={"ID":"b079fa40-b11a-409f-a006-b8e7454e1b89","Type":"ContainerStarted","Data":"f1a58918df29c5b0339ea093de4b6e280ec4b17a7cdf3924fa3ad562983cf31f"}
Oct 31 01:48:11.063089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:11.063030    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8" event={"ID":"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb","Type":"ContainerStarted","Data":"c7a2bff4750647951203289a239911b1d65fb426df41d12a0944b3d2d75e3dea"}
Oct 31 01:48:11.063248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:11.063113    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8" event={"ID":"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb","Type":"ContainerStarted","Data":"df6a3c43f171f4bb3b4dd43b4983327f5530eae710ae4a46869f224d1a1fdd73"}
Oct 31 01:48:11.086719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:11.086676    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8" podStartSLOduration=2.086662642 podStartE2EDuration="2.086662642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:11.081952245 +0000 UTC m=+1649.030600631" watchObservedRunningTime="2026-10-31 01:48:11.086662642 +0000 UTC m=+1649.035311034"
Oct 31 01:48:11.875415 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:11.875379    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:48:12.073281 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:12.073197    5293 generic.go:356] "Generic (PLEG): container finished" podID="b079fa40-b11a-409f-a006-b8e7454e1b89" containerID="14ada5cfa422255dbeedfa866d053c75c12c8128d1f553ea643c716f48818af9" exitCode=0
Oct 31 01:48:12.073447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:12.073276    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps" event={"ID":"b079fa40-b11a-409f-a006-b8e7454e1b89","Type":"ContainerDied","Data":"14ada5cfa422255dbeedfa866d053c75c12c8128d1f553ea643c716f48818af9"}
Oct 31 01:48:12.075010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:12.074992    5293 generic.go:356] "Generic (PLEG): container finished" podID="1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb" containerID="c7a2bff4750647951203289a239911b1d65fb426df41d12a0944b3d2d75e3dea" exitCode=0
Oct 31 01:48:12.075146 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:12.075070    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8" event={"ID":"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb","Type":"ContainerDied","Data":"c7a2bff4750647951203289a239911b1d65fb426df41d12a0944b3d2d75e3dea"}
Oct 31 01:48:13.470269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.470246    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:13.472797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.472777    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:13.586989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.586957    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-openshift-ca\") pod \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") "
Oct 31 01:48:13.587197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.587018    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b079fa40-b11a-409f-a006-b8e7454e1b89-kube-api-access-kkhkx\" (UniqueName: \"kubernetes.io/projected/b079fa40-b11a-409f-a006-b8e7454e1b89-kube-api-access-kkhkx\") pod \"b079fa40-b11a-409f-a006-b8e7454e1b89\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") "
Oct 31 01:48:13.587197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.587067    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-kube-api-access-89x8c\" (UniqueName: \"kubernetes.io/projected/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-kube-api-access-89x8c\") pod \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") "
Oct 31 01:48:13.587197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.587156    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-xks-ca\") pod \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\" (UID: \"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb\") "
Oct 31 01:48:13.587377 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.587297    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-xks-ca\") pod \"b079fa40-b11a-409f-a006-b8e7454e1b89\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") "
Oct 31 01:48:13.587377 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.587303    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-openshift-ca" pod "1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb" (UID: "1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:13.587545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.587446    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-openshift-ca\") pod \"b079fa40-b11a-409f-a006-b8e7454e1b89\" (UID: \"b079fa40-b11a-409f-a006-b8e7454e1b89\") "
Oct 31 01:48:13.587996 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.587870    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-openshift-ca" pod "b079fa40-b11a-409f-a006-b8e7454e1b89" (UID: "b079fa40-b11a-409f-a006-b8e7454e1b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:13.588667 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.588640    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:13.588769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.588671    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:13.589445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.589414    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-kube-api-access-89x8c" pod "1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb" (UID: "1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb"). InnerVolumeSpecName "kube-api-access-89x8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:13.589552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.589460    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b079fa40-b11a-409f-a006-b8e7454e1b89-kube-api-access-kkhkx" pod "b079fa40-b11a-409f-a006-b8e7454e1b89" (UID: "b079fa40-b11a-409f-a006-b8e7454e1b89"). InnerVolumeSpecName "kube-api-access-kkhkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:13.589552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.589492    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-xks-ca" pod "1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb" (UID: "1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:48:13.589646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.589624    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-xks-ca" pod "b079fa40-b11a-409f-a006-b8e7454e1b89" (UID: "b079fa40-b11a-409f-a006-b8e7454e1b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:48:13.695073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.694714    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b079fa40-b11a-409f-a006-b8e7454e1b89-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:13.695073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.694753    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhkx\" (UniqueName: \"kubernetes.io/projected/b079fa40-b11a-409f-a006-b8e7454e1b89-kube-api-access-kkhkx\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:13.695073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.694769    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89x8c\" (UniqueName: \"kubernetes.io/projected/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-kube-api-access-89x8c\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:13.695073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.694781    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:13.810062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.809813    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:13.830066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:13.822704    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:14.211438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:14.210857    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"
Oct 31 01:48:14.215471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:14.215428    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"
Oct 31 01:48:14.218754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:14.218718    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8" event={"ID":"1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb","Type":"ContainerDied","Data":"df6a3c43f171f4bb3b4dd43b4983327f5530eae710ae4a46869f224d1a1fdd73"}
Oct 31 01:48:14.218888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:14.218782    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6a3c43f171f4bb3b4dd43b4983327f5530eae710ae4a46869f224d1a1fdd73"
Oct 31 01:48:14.218888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:14.218800    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps" event={"ID":"b079fa40-b11a-409f-a006-b8e7454e1b89","Type":"ContainerDied","Data":"f1a58918df29c5b0339ea093de4b6e280ec4b17a7cdf3924fa3ad562983cf31f"}
Oct 31 01:48:14.218888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:14.218824    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1a58918df29c5b0339ea093de4b6e280ec4b17a7cdf3924fa3ad562983cf31f"
Oct 31 01:48:19.689301 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:19.689265    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"]
Oct 31 01:48:19.720064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:19.705698    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"]
Oct 31 01:48:20.241064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:20.220817    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"]
Oct 31 01:48:20.241064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:20.220900    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"]
Oct 31 01:48:20.694591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:20.694560    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"]
Oct 31 01:48:20.751073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:20.744642    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"]
Oct 31 01:48:22.929671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:22.929639    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"]
Oct 31 01:48:22.935074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:22.935032    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb5756cb-plfn8"]
Oct 31 01:48:24.175327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:24.175296    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb" path="/var/lib/kubelet/pods/1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb/volumes"
Oct 31 01:48:24.691562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:24.691509    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db" podUID="55aa2c7b-086c-4cae-a9f2-76b1293b9eb2" containerName="payload-processing" containerID="cri-o://3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d" gracePeriod=25
Oct 31 01:48:24.706761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:24.706701    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4" podUID="e6fc28ff-b89f-4c52-95e8-80746d2d7f9f" containerName="payload-pre-processing" containerID="cri-o://f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5" gracePeriod=25
Oct 31 01:48:25.225920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.225845    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t" podUID="c02a453e-4093-410d-9757-c2e3272dab99" containerName="payload-processing" containerID="cri-o://5abfdad8314d2fe4b55d3fd4759d161d68d7fbc4901518153ea82780d0007bea" gracePeriod=25
Oct 31 01:48:25.235243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.226415    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d" podUID="5f65019d-5f02-4a97-9a6e-39e4703f588c" containerName="payload-pre-processing" containerID="cri-o://c3caf70b4b75ae58f20c22551a96eeebb1cc296607b91561bea0c5ce2c47ef4c" gracePeriod=25
Oct 31 01:48:25.337261 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:25.331724    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02a453e_4093_410d_9757_c2e3272dab99.slice/crio-conmon-5abfdad8314d2fe4b55d3fd4759d161d68d7fbc4901518153ea82780d0007bea.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f65019d_5f02_4a97_9a6e_39e4703f588c.slice/crio-c3caf70b4b75ae58f20c22551a96eeebb1cc296607b91561bea0c5ce2c47ef4c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02a453e_4093_410d_9757_c2e3272dab99.slice/crio-5abfdad8314d2fe4b55d3fd4759d161d68d7fbc4901518153ea82780d0007bea.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:48:25.470335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.469946    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:48:25.516022 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.515249    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:48:25.545085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.542878    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-plugins-config-volume\") pod \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\" (UID: \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\") "
Oct 31 01:48:25.545085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.542964    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-plugins-config-volume\") pod \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\" (UID: \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\") "
Oct 31 01:48:25.545085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.543216    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-kube-api-access-2tpsf\" (UniqueName: \"kubernetes.io/projected/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-kube-api-access-2tpsf\") pod \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\" (UID: \"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f\") "
Oct 31 01:48:25.545085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.543281    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-kube-api-access-x5zw9\" (UniqueName: \"kubernetes.io/projected/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-kube-api-access-x5zw9\") pod \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\" (UID: \"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2\") "
Oct 31 01:48:25.545085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.544769    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-plugins-config-volume" pod "e6fc28ff-b89f-4c52-95e8-80746d2d7f9f" (UID: "e6fc28ff-b89f-4c52-95e8-80746d2d7f9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:25.545477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.545424    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-plugins-config-volume" pod "55aa2c7b-086c-4cae-a9f2-76b1293b9eb2" (UID: "55aa2c7b-086c-4cae-a9f2-76b1293b9eb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:25.552443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.552276    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-kube-api-access-2tpsf" pod "e6fc28ff-b89f-4c52-95e8-80746d2d7f9f" (UID: "e6fc28ff-b89f-4c52-95e8-80746d2d7f9f"). InnerVolumeSpecName "kube-api-access-2tpsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:25.552859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.552834    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-kube-api-access-x5zw9" pod "55aa2c7b-086c-4cae-a9f2-76b1293b9eb2" (UID: "55aa2c7b-086c-4cae-a9f2-76b1293b9eb2"). InnerVolumeSpecName "kube-api-access-x5zw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:25.581439 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.581404    5293 generic.go:356] "Generic (PLEG): container finished" podID="c02a453e-4093-410d-9757-c2e3272dab99" containerID="5abfdad8314d2fe4b55d3fd4759d161d68d7fbc4901518153ea82780d0007bea" exitCode=0
Oct 31 01:48:25.581624 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.581605    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t" event={"ID":"c02a453e-4093-410d-9757-c2e3272dab99","Type":"ContainerDied","Data":"5abfdad8314d2fe4b55d3fd4759d161d68d7fbc4901518153ea82780d0007bea"}
Oct 31 01:48:25.583396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.583369    5293 generic.go:356] "Generic (PLEG): container finished" podID="55aa2c7b-086c-4cae-a9f2-76b1293b9eb2" containerID="3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d" exitCode=0
Oct 31 01:48:25.583523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.583501    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"
Oct 31 01:48:25.584090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.584060    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db" event={"ID":"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2","Type":"ContainerDied","Data":"3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d"}
Oct 31 01:48:25.584224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.584119    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db" event={"ID":"55aa2c7b-086c-4cae-a9f2-76b1293b9eb2","Type":"ContainerDied","Data":"787220a4eb9a512e4b52b62e4dde07ceda6abb54df052e115643a374bf88aaff"}
Oct 31 01:48:25.584224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.584180    5293 scope.go:122] "RemoveContainer" containerID="3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d"
Oct 31 01:48:25.586852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.586827    5293 generic.go:356] "Generic (PLEG): container finished" podID="e6fc28ff-b89f-4c52-95e8-80746d2d7f9f" containerID="f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5" exitCode=0
Oct 31 01:48:25.586989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.586970    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4" event={"ID":"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f","Type":"ContainerDied","Data":"f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5"}
Oct 31 01:48:25.587077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.587019    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4" event={"ID":"e6fc28ff-b89f-4c52-95e8-80746d2d7f9f","Type":"ContainerDied","Data":"1f34dbeb80beab7c35c37504c4704653d2493c6785bd45ec65e87a34d898ea80"}
Oct 31 01:48:25.587165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.587145    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"
Oct 31 01:48:25.622011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.621792    5293 generic.go:356] "Generic (PLEG): container finished" podID="5f65019d-5f02-4a97-9a6e-39e4703f588c" containerID="c3caf70b4b75ae58f20c22551a96eeebb1cc296607b91561bea0c5ce2c47ef4c" exitCode=0
Oct 31 01:48:25.622011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.621956    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d" event={"ID":"5f65019d-5f02-4a97-9a6e-39e4703f588c","Type":"ContainerDied","Data":"c3caf70b4b75ae58f20c22551a96eeebb1cc296607b91561bea0c5ce2c47ef4c"}
Oct 31 01:48:25.636517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.636404    5293 scope.go:122] "RemoveContainer" containerID="3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d"
Oct 31 01:48:25.637062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.636942    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"]
Oct 31 01:48:25.638105 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:25.637508    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d\": container with ID starting with 3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d not found: ID does not exist" containerID="3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d"
Oct 31 01:48:25.638105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.637538    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d"} err="failed to get container status \"3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d\": rpc error: code = NotFound desc = could not find container \"3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d\": container with ID starting with 3b636fd97836604fd36158e348f0c9752abe33215548f61865a191cb992d7d5d not found: ID does not exist"
Oct 31 01:48:25.638105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.637563    5293 scope.go:122] "RemoveContainer" containerID="f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5"
Oct 31 01:48:25.645593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.643892    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-pf5db"]
Oct 31 01:48:25.645593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.645310    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tpsf\" (UniqueName: \"kubernetes.io/projected/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-kube-api-access-2tpsf\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:25.645593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.645329    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5zw9\" (UniqueName: \"kubernetes.io/projected/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-kube-api-access-x5zw9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:25.645593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.645339    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:25.645593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.645349    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:25.657341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.656616    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"]
Oct 31 01:48:25.664678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.664650    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-sxfk4"]
Oct 31 01:48:25.668716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.668182    5293 scope.go:122] "RemoveContainer" containerID="f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5"
Oct 31 01:48:25.670145 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:25.669837    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5\": container with ID starting with f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5 not found: ID does not exist" containerID="f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5"
Oct 31 01:48:25.670145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.669894    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5"} err="failed to get container status \"f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5\": rpc error: code = NotFound desc = could not find container \"f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5\": container with ID starting with f0ed131cb311370e348b8d2e819f7e9b60c76b1c5b949a00d4fa4f58af6d1fd5 not found: ID does not exist"
Oct 31 01:48:25.686064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.685856    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:25.694999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.694960    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5" podUID="715e9e29-dbe7-4866-ba65-672cf2a06267" containerName="payload-processing" containerID="cri-o://39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3" gracePeriod=25
Oct 31 01:48:25.746975 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.745313    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc" podUID="16679091-1da3-428d-ba93-611ba19cd055" containerName="payload-pre-processing" containerID="cri-o://b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1" gracePeriod=25
Oct 31 01:48:25.847404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.847363    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f65019d-5f02-4a97-9a6e-39e4703f588c-kube-api-access-x5ggg\" (UniqueName: \"kubernetes.io/projected/5f65019d-5f02-4a97-9a6e-39e4703f588c-kube-api-access-x5ggg\") pod \"5f65019d-5f02-4a97-9a6e-39e4703f588c\" (UID: \"5f65019d-5f02-4a97-9a6e-39e4703f588c\") "
Oct 31 01:48:25.847604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.847540    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f65019d-5f02-4a97-9a6e-39e4703f588c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f65019d-5f02-4a97-9a6e-39e4703f588c-plugins-config-volume\") pod \"5f65019d-5f02-4a97-9a6e-39e4703f588c\" (UID: \"5f65019d-5f02-4a97-9a6e-39e4703f588c\") "
Oct 31 01:48:25.848115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.848028    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f65019d-5f02-4a97-9a6e-39e4703f588c-plugins-config-volume" pod "5f65019d-5f02-4a97-9a6e-39e4703f588c" (UID: "5f65019d-5f02-4a97-9a6e-39e4703f588c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:25.848429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.848407    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f65019d-5f02-4a97-9a6e-39e4703f588c-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:25.850314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.850283    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f65019d-5f02-4a97-9a6e-39e4703f588c-kube-api-access-x5ggg" pod "5f65019d-5f02-4a97-9a6e-39e4703f588c" (UID: "5f65019d-5f02-4a97-9a6e-39e4703f588c"). InnerVolumeSpecName "kube-api-access-x5ggg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:25.852820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.852792    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"]
Oct 31 01:48:25.853578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853558    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b079fa40-b11a-409f-a006-b8e7454e1b89" containerName="revoke-keys"
Oct 31 01:48:25.853578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853579    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b079fa40-b11a-409f-a006-b8e7454e1b89" containerName="revoke-keys"
Oct 31 01:48:25.853721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853616    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5f65019d-5f02-4a97-9a6e-39e4703f588c" containerName="payload-pre-processing"
Oct 31 01:48:25.853721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853624    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f65019d-5f02-4a97-9a6e-39e4703f588c" containerName="payload-pre-processing"
Oct 31 01:48:25.853721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853634    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb" containerName="revoke-keys"
Oct 31 01:48:25.853721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853641    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb" containerName="revoke-keys"
Oct 31 01:48:25.853721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853663    5293 cpu_manager.go:414] "containerMap: removing container" podUID="55aa2c7b-086c-4cae-a9f2-76b1293b9eb2" containerName="payload-processing"
Oct 31 01:48:25.853721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853671    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55aa2c7b-086c-4cae-a9f2-76b1293b9eb2" containerName="payload-processing"
Oct 31 01:48:25.853721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853684    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e6fc28ff-b89f-4c52-95e8-80746d2d7f9f" containerName="payload-pre-processing"
Oct 31 01:48:25.853721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853692    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fc28ff-b89f-4c52-95e8-80746d2d7f9f" containerName="payload-pre-processing"
Oct 31 01:48:25.854140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853901    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b079fa40-b11a-409f-a006-b8e7454e1b89" containerName="revoke-keys"
Oct 31 01:48:25.854140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853920    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f65019d-5f02-4a97-9a6e-39e4703f588c" containerName="payload-pre-processing"
Oct 31 01:48:25.854140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853929    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa1c93e-2e8a-40bc-b5cb-f7e8236c5fcb" containerName="revoke-keys"
Oct 31 01:48:25.854140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853942    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="55aa2c7b-086c-4cae-a9f2-76b1293b9eb2" containerName="payload-processing"
Oct 31 01:48:25.854140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.853951    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fc28ff-b89f-4c52-95e8-80746d2d7f9f" containerName="payload-pre-processing"
Oct 31 01:48:25.858091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.857980    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:25.867006 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.866950    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f0694-dockercfg-nlb72\""
Oct 31 01:48:25.867603 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.867008    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9f0694\""
Oct 31 01:48:25.875235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.875204    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"]
Oct 31 01:48:25.883870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.883541    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:25.953774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.950261    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02a453e-4093-410d-9757-c2e3272dab99-kube-api-access-jwq7x\" (UniqueName: \"kubernetes.io/projected/c02a453e-4093-410d-9757-c2e3272dab99-kube-api-access-jwq7x\") pod \"c02a453e-4093-410d-9757-c2e3272dab99\" (UID: \"c02a453e-4093-410d-9757-c2e3272dab99\") "
Oct 31 01:48:25.953774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.950853    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02a453e-4093-410d-9757-c2e3272dab99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a453e-4093-410d-9757-c2e3272dab99-plugins-config-volume\") pod \"c02a453e-4093-410d-9757-c2e3272dab99\" (UID: \"c02a453e-4093-410d-9757-c2e3272dab99\") "
Oct 31 01:48:25.953774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.951442    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d85d5-af98-4405-b040-dedf3c75e472-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb\" (UID: \"064d85d5-af98-4405-b040-dedf3c75e472\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:25.953774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.951544    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02a453e-4093-410d-9757-c2e3272dab99-plugins-config-volume" pod "c02a453e-4093-410d-9757-c2e3272dab99" (UID: "c02a453e-4093-410d-9757-c2e3272dab99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:25.953774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.951741    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m66m9\" (UniqueName: \"kubernetes.io/projected/064d85d5-af98-4405-b040-dedf3c75e472-kube-api-access-m66m9\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb\" (UID: \"064d85d5-af98-4405-b040-dedf3c75e472\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:25.953774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.951900    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5ggg\" (UniqueName: \"kubernetes.io/projected/5f65019d-5f02-4a97-9a6e-39e4703f588c-kube-api-access-x5ggg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:25.953774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.951920    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a453e-4093-410d-9757-c2e3272dab99-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:25.953774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:25.953101    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02a453e-4093-410d-9757-c2e3272dab99-kube-api-access-jwq7x" pod "c02a453e-4093-410d-9757-c2e3272dab99" (UID: "c02a453e-4093-410d-9757-c2e3272dab99"). InnerVolumeSpecName "kube-api-access-jwq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:26.131219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.084762    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d85d5-af98-4405-b040-dedf3c75e472-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb\" (UID: \"064d85d5-af98-4405-b040-dedf3c75e472\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:26.131219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.084963    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m66m9\" (UniqueName: \"kubernetes.io/projected/064d85d5-af98-4405-b040-dedf3c75e472-kube-api-access-m66m9\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb\" (UID: \"064d85d5-af98-4405-b040-dedf3c75e472\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:26.131219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.085180    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwq7x\" (UniqueName: \"kubernetes.io/projected/c02a453e-4093-410d-9757-c2e3272dab99-kube-api-access-jwq7x\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:26.137106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.131932    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d85d5-af98-4405-b040-dedf3c75e472-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb\" (UID: \"064d85d5-af98-4405-b040-dedf3c75e472\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:26.310128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.299730    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m66m9\" (UniqueName: \"kubernetes.io/projected/064d85d5-af98-4405-b040-dedf3c75e472-kube-api-access-m66m9\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb\" (UID: \"064d85d5-af98-4405-b040-dedf3c75e472\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:26.310128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.307208    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:48:26.332890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.332843    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55aa2c7b-086c-4cae-a9f2-76b1293b9eb2" path="/var/lib/kubelet/pods/55aa2c7b-086c-4cae-a9f2-76b1293b9eb2/volumes"
Oct 31 01:48:26.333885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.333785    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6fc28ff-b89f-4c52-95e8-80746d2d7f9f" path="/var/lib/kubelet/pods/e6fc28ff-b89f-4c52-95e8-80746d2d7f9f/volumes"
Oct 31 01:48:26.439606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.438327    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715e9e29-dbe7-4866-ba65-672cf2a06267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e9e29-dbe7-4866-ba65-672cf2a06267-plugins-config-volume\") pod \"715e9e29-dbe7-4866-ba65-672cf2a06267\" (UID: \"715e9e29-dbe7-4866-ba65-672cf2a06267\") "
Oct 31 01:48:26.439606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.438385    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715e9e29-dbe7-4866-ba65-672cf2a06267-kube-api-access-qp8jq\" (UniqueName: \"kubernetes.io/projected/715e9e29-dbe7-4866-ba65-672cf2a06267-kube-api-access-qp8jq\") pod \"715e9e29-dbe7-4866-ba65-672cf2a06267\" (UID: \"715e9e29-dbe7-4866-ba65-672cf2a06267\") "
Oct 31 01:48:26.489220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.489168    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715e9e29-dbe7-4866-ba65-672cf2a06267-plugins-config-volume" pod "715e9e29-dbe7-4866-ba65-672cf2a06267" (UID: "715e9e29-dbe7-4866-ba65-672cf2a06267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:26.493966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.493711    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:26.556426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.556380    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715e9e29-dbe7-4866-ba65-672cf2a06267-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:26.568440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.565908    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715e9e29-dbe7-4866-ba65-672cf2a06267-kube-api-access-qp8jq" pod "715e9e29-dbe7-4866-ba65-672cf2a06267" (UID: "715e9e29-dbe7-4866-ba65-672cf2a06267"). InnerVolumeSpecName "kube-api-access-qp8jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:26.587526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.587502    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:48:26.638095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.638061    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d" event={"ID":"5f65019d-5f02-4a97-9a6e-39e4703f588c","Type":"ContainerDied","Data":"9e54a8d505eefda621bdbfd8b2029eb16e866078891e8a356283dd897cad5c01"}
Oct 31 01:48:26.638264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.638133    5293 scope.go:122] "RemoveContainer" containerID="c3caf70b4b75ae58f20c22551a96eeebb1cc296607b91561bea0c5ce2c47ef4c"
Oct 31 01:48:26.638264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.638253    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"
Oct 31 01:48:26.642953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.642573    5293 generic.go:356] "Generic (PLEG): container finished" podID="16679091-1da3-428d-ba93-611ba19cd055" containerID="b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1" exitCode=0
Oct 31 01:48:26.643397 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.643313    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"
Oct 31 01:48:26.643627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.643573    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc" event={"ID":"16679091-1da3-428d-ba93-611ba19cd055","Type":"ContainerDied","Data":"b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1"}
Oct 31 01:48:26.643627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.643622    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc" event={"ID":"16679091-1da3-428d-ba93-611ba19cd055","Type":"ContainerDied","Data":"5ea9b38b078c33a2f778c86729f82fe9ea566d6d28c64fb9e512a6662a01fad9"}
Oct 31 01:48:26.648520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.647741    5293 generic.go:356] "Generic (PLEG): container finished" podID="715e9e29-dbe7-4866-ba65-672cf2a06267" containerID="39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3" exitCode=0
Oct 31 01:48:26.648520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.647794    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5" event={"ID":"715e9e29-dbe7-4866-ba65-672cf2a06267","Type":"ContainerDied","Data":"39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3"}
Oct 31 01:48:26.648520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.647825    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5" event={"ID":"715e9e29-dbe7-4866-ba65-672cf2a06267","Type":"ContainerDied","Data":"158688e030a00fb724eebe85b44818d33191bae771a4641620c9c3dc39e7a2f2"}
Oct 31 01:48:26.648520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.647879    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"
Oct 31 01:48:26.657777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.657747    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16679091-1da3-428d-ba93-611ba19cd055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16679091-1da3-428d-ba93-611ba19cd055-plugins-config-volume\") pod \"16679091-1da3-428d-ba93-611ba19cd055\" (UID: \"16679091-1da3-428d-ba93-611ba19cd055\") "
Oct 31 01:48:26.657921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.657862    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16679091-1da3-428d-ba93-611ba19cd055-kube-api-access-dhwbv\" (UniqueName: \"kubernetes.io/projected/16679091-1da3-428d-ba93-611ba19cd055-kube-api-access-dhwbv\") pod \"16679091-1da3-428d-ba93-611ba19cd055\" (UID: \"16679091-1da3-428d-ba93-611ba19cd055\") "
Oct 31 01:48:26.658582 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.658527    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8jq\" (UniqueName: \"kubernetes.io/projected/715e9e29-dbe7-4866-ba65-672cf2a06267-kube-api-access-qp8jq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:26.660488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.660449    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16679091-1da3-428d-ba93-611ba19cd055-plugins-config-volume" pod "16679091-1da3-428d-ba93-611ba19cd055" (UID: "16679091-1da3-428d-ba93-611ba19cd055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:26.661689 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.661659    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16679091-1da3-428d-ba93-611ba19cd055-kube-api-access-dhwbv" pod "16679091-1da3-428d-ba93-611ba19cd055" (UID: "16679091-1da3-428d-ba93-611ba19cd055"). InnerVolumeSpecName "kube-api-access-dhwbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:26.662493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.662469    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"
Oct 31 01:48:26.663075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.663029    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t" event={"ID":"c02a453e-4093-410d-9757-c2e3272dab99","Type":"ContainerDied","Data":"af710856ff0f1c0be8481b67114f5bff985cc29871a70ab572234fb0ce957a49"}
Oct 31 01:48:26.664123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.664104    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"]
Oct 31 01:48:26.666962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.666944    5293 scope.go:122] "RemoveContainer" containerID="b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1"
Oct 31 01:48:26.670562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.670542    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-bb4884467-74p5d"]
Oct 31 01:48:26.681224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.681153    5293 scope.go:122] "RemoveContainer" containerID="b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1"
Oct 31 01:48:26.681590 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:26.681451    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1\": container with ID starting with b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1 not found: ID does not exist" containerID="b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1"
Oct 31 01:48:26.681590 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.681490    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1"} err="failed to get container status \"b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1\": rpc error: code = NotFound desc = could not find container \"b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1\": container with ID starting with b12a9a69ed46f1cad78ddbbb52749b9936d33ab0222fe9dbebc59ab6f23b06b1 not found: ID does not exist"
Oct 31 01:48:26.681590 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.681513    5293 scope.go:122] "RemoveContainer" containerID="39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3"
Oct 31 01:48:26.686188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.686170    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"]
Oct 31 01:48:26.692139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.692094    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-65dc4c599c-n964t"]
Oct 31 01:48:26.698750 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.698603    5293 scope.go:122] "RemoveContainer" containerID="39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3"
Oct 31 01:48:26.701062 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:26.699112    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3\": container with ID starting with 39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3 not found: ID does not exist" containerID="39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3"
Oct 31 01:48:26.701062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.699150    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3"} err="failed to get container status \"39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3\": rpc error: code = NotFound desc = could not find container \"39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3\": container with ID starting with 39ce198c1faecf6b6085358838ea22809db3eea4ff1122564ab11990b21cead3 not found: ID does not exist"
Oct 31 01:48:26.701062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.699174    5293 scope.go:122] "RemoveContainer" containerID="5abfdad8314d2fe4b55d3fd4759d161d68d7fbc4901518153ea82780d0007bea"
Oct 31 01:48:26.706851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.706801    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"]
Oct 31 01:48:26.713711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.713693    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-50ac12-776f46ff5-rrkq5"]
Oct 31 01:48:26.760342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.760318    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16679091-1da3-428d-ba93-611ba19cd055-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:26.760342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.760340    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwbv\" (UniqueName: \"kubernetes.io/projected/16679091-1da3-428d-ba93-611ba19cd055-kube-api-access-dhwbv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:26.859225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.859199    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"]
Oct 31 01:48:26.859626 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:48:26.859607    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064d85d5_af98_4405_b040_dedf3c75e472.slice/crio-4dab8851b7aa307e8cf2bad46fb018310d96e7cc8516b5b1391a00d4cbecc30b WatchSource:0}: Error finding container 4dab8851b7aa307e8cf2bad46fb018310d96e7cc8516b5b1391a00d4cbecc30b: Status 404 returned error can't find the container with id 4dab8851b7aa307e8cf2bad46fb018310d96e7cc8516b5b1391a00d4cbecc30b
Oct 31 01:48:26.861711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.861692    5293 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:48:26.967815 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.967788    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"]
Oct 31 01:48:26.973404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:26.973378    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-50ac12-56dfcf7975-lr4fc"]
Oct 31 01:48:27.025246 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:27.025210    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"]
Oct 31 01:48:27.030149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:27.030126    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936429de-mt8zc"]
Oct 31 01:48:27.683498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:27.683461    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb" event={"ID":"064d85d5-af98-4405-b040-dedf3c75e472","Type":"ContainerStarted","Data":"4dab8851b7aa307e8cf2bad46fb018310d96e7cc8516b5b1391a00d4cbecc30b"}
Oct 31 01:48:28.177609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:28.177572    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de343be-256a-4503-a8b4-55adc17989a0" path="/var/lib/kubelet/pods/0de343be-256a-4503-a8b4-55adc17989a0/volumes"
Oct 31 01:48:28.178259 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:28.178235    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16679091-1da3-428d-ba93-611ba19cd055" path="/var/lib/kubelet/pods/16679091-1da3-428d-ba93-611ba19cd055/volumes"
Oct 31 01:48:28.178771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:28.178745    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f65019d-5f02-4a97-9a6e-39e4703f588c" path="/var/lib/kubelet/pods/5f65019d-5f02-4a97-9a6e-39e4703f588c/volumes"
Oct 31 01:48:28.179542 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:28.179522    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715e9e29-dbe7-4866-ba65-672cf2a06267" path="/var/lib/kubelet/pods/715e9e29-dbe7-4866-ba65-672cf2a06267/volumes"
Oct 31 01:48:28.180010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:28.179993    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02a453e-4093-410d-9757-c2e3272dab99" path="/var/lib/kubelet/pods/c02a453e-4093-410d-9757-c2e3272dab99/volumes"
Oct 31 01:48:28.888300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:28.881424    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"]
Oct 31 01:48:28.889079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:28.888891    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-50ac12-f2lps"]
Oct 31 01:48:29.800585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:29.800548    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb" event={"ID":"064d85d5-af98-4405-b040-dedf3c75e472","Type":"ContainerStarted","Data":"b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054"}
Oct 31 01:48:29.800786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:29.800716    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:29.819878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:29.819809    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb" podStartSLOduration=3.129769923 podStartE2EDuration="4.819793164s" totalImagesPullingTime="1.690023241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:29.819073514 +0000 UTC m=+1667.767721906" watchObservedRunningTime="2026-10-31 01:48:29.819793164 +0000 UTC m=+1667.768441560"
Oct 31 01:48:30.174476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:30.174449    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b079fa40-b11a-409f-a006-b8e7454e1b89" path="/var/lib/kubelet/pods/b079fa40-b11a-409f-a006-b8e7454e1b89/volumes"
Oct 31 01:48:31.224647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:31.224612    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:48:36.071276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.071240    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"]
Oct 31 01:48:36.071746 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.071499    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy" containerID="cri-o://9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29" gracePeriod=30
Oct 31 01:48:36.077102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.077083    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"]
Oct 31 01:48:36.085719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.085694    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f01db46-88gqj"]
Oct 31 01:48:36.514065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.503477    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6022b938-15c3-426e-8249-0aa49b38bfa0" path="/var/lib/kubelet/pods/6022b938-15c3-426e-8249-0aa49b38bfa0/volumes"
Oct 31 01:48:36.575124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.575093    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"]
Oct 31 01:48:36.576324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.576300    5293 cpu_manager.go:414] "containerMap: removing container" podUID="715e9e29-dbe7-4866-ba65-672cf2a06267" containerName="payload-processing"
Oct 31 01:48:36.576449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.576438    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715e9e29-dbe7-4866-ba65-672cf2a06267" containerName="payload-processing"
Oct 31 01:48:36.576558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.576549    5293 cpu_manager.go:414] "containerMap: removing container" podUID="16679091-1da3-428d-ba93-611ba19cd055" containerName="payload-pre-processing"
Oct 31 01:48:36.576625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.576615    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16679091-1da3-428d-ba93-611ba19cd055" containerName="payload-pre-processing"
Oct 31 01:48:36.576731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.576721    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c02a453e-4093-410d-9757-c2e3272dab99" containerName="payload-processing"
Oct 31 01:48:36.576803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.576794    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02a453e-4093-410d-9757-c2e3272dab99" containerName="payload-processing"
Oct 31 01:48:36.577055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.577026    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="715e9e29-dbe7-4866-ba65-672cf2a06267" containerName="payload-processing"
Oct 31 01:48:36.577164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.577153    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="16679091-1da3-428d-ba93-611ba19cd055" containerName="payload-pre-processing"
Oct 31 01:48:36.577250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.577239    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02a453e-4093-410d-9757-c2e3272dab99" containerName="payload-processing"
Oct 31 01:48:36.608075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.590629    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"]
Oct 31 01:48:36.608075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.590760    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.609456 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.609420    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:48:36.609597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.609529    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:48:36.609663 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.609641    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:48:36.609748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.609727    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:48:36.693651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.693605    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twwd8\" (UniqueName: \"kubernetes.io/projected/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-kube-api-access-twwd8\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.693841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.693663    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.693841 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.693776    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.795172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.795067    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twwd8\" (UniqueName: \"kubernetes.io/projected/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-kube-api-access-twwd8\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.795172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.795103    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.795443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.795177    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.795885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.795860    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.834666 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.834614    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:36.835987 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:36.835945    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twwd8\" (UniqueName: \"kubernetes.io/projected/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-kube-api-access-twwd8\") pod \"maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:37.049946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:37.049852    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:37.468246 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:37.466713    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"]
Oct 31 01:48:37.474301 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:48:37.474268    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8513c054_e1f0_45c8_8f2c_1c38431ec5fc.slice/crio-ef64089b2dba411ed7d8898e8f4abd07026d8325ee44beba723978b6effbc020 WatchSource:0}: Error finding container ef64089b2dba411ed7d8898e8f4abd07026d8325ee44beba723978b6effbc020: Status 404 returned error can't find the container with id ef64089b2dba411ed7d8898e8f4abd07026d8325ee44beba723978b6effbc020
Oct 31 01:48:37.983284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:37.983247    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb" event={"ID":"8513c054-e1f0-45c8-8f2c-1c38431ec5fc","Type":"ContainerStarted","Data":"76e10329a5e9e5e249e6e7f6c3e0c57a283aac7a7d3998c7039b6a1151585207"}
Oct 31 01:48:37.983533 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:37.983373    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb" event={"ID":"8513c054-e1f0-45c8-8f2c-1c38431ec5fc","Type":"ContainerStarted","Data":"ef64089b2dba411ed7d8898e8f4abd07026d8325ee44beba723978b6effbc020"}
Oct 31 01:48:38.015065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:38.010817    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb" podStartSLOduration=2.010799047 podStartE2EDuration="2.010799047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:38.005907178 +0000 UTC m=+1675.954555569" watchObservedRunningTime="2026-10-31 01:48:38.010799047 +0000 UTC m=+1675.959447456"
Oct 31 01:48:38.924864 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:38.924822    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"]
Oct 31 01:48:38.931204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:38.931168    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:38.934614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:38.934573    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb5756cb-openshift-default-dockercfg-twxqj\""
Oct 31 01:48:38.942189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:38.942165    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"]
Oct 31 01:48:38.995435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:38.995402    5293 generic.go:356] "Generic (PLEG): container finished" podID="8513c054-e1f0-45c8-8f2c-1c38431ec5fc" containerID="76e10329a5e9e5e249e6e7f6c3e0c57a283aac7a7d3998c7039b6a1151585207" exitCode=0
Oct 31 01:48:38.995622 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:38.995485    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb" event={"ID":"8513c054-e1f0-45c8-8f2c-1c38431ec5fc","Type":"ContainerDied","Data":"76e10329a5e9e5e249e6e7f6c3e0c57a283aac7a7d3998c7039b6a1151585207"}
Oct 31 01:48:39.015924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.015876    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-credential-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.016129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.016033    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-certs\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.016224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.016141    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840bd7db-af25-418d-964a-9640bb1bff43-istiod-ca-cert\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.016224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.016168    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4xk\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-kube-api-access-ml4xk\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.016224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.016214    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-data\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.016398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.016241    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-istio-token\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.016398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.016317    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840bd7db-af25-418d-964a-9640bb1bff43-istio-podinfo\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.016398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.016352    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.016398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.016380    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-envoy\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.117859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.117822    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-credential-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.117913    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-certs\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.117986    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840bd7db-af25-418d-964a-9640bb1bff43-istiod-ca-cert\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118009    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4xk\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-kube-api-access-ml4xk\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118094    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-data\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118120    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-istio-token\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118199    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840bd7db-af25-418d-964a-9640bb1bff43-istio-podinfo\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118237    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118264    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-envoy\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118337    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-credential-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118397    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-certs\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118663    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-data\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118739    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840bd7db-af25-418d-964a-9640bb1bff43-istiod-ca-cert\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.118932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.118910    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.121226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.121203    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-envoy\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.121702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.121680    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840bd7db-af25-418d-964a-9640bb1bff43-istio-podinfo\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.130986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.130930    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-istio-token\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.131174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.131154    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4xk\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-kube-api-access-ml4xk\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.256269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.256177    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:39.614891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.614803    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"]
Oct 31 01:48:39.643802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.643751    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:39.643952 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.643855    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:39.643952 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:39.643899    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:40.011318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.011276    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s" event={"ID":"840bd7db-af25-418d-964a-9640bb1bff43","Type":"ContainerStarted","Data":"b1580b64a145bec3ef503318d78bc1297acde8f58392c78b7494e728ea3834a7"}
Oct 31 01:48:40.011809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.011348    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s" event={"ID":"840bd7db-af25-418d-964a-9640bb1bff43","Type":"ContainerStarted","Data":"99317d1a79e47e63434f938689e4961d9d1b1e6fc5985e899aca71bf2ae315d9"}
Oct 31 01:48:40.048985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.048904    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s" podStartSLOduration=2.048885121 podStartE2EDuration="2.048885121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:40.042980544 +0000 UTC m=+1677.991628936" watchObservedRunningTime="2026-10-31 01:48:40.048885121 +0000 UTC m=+1677.997533514"
Oct 31 01:48:40.406340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.385615    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:40.463452 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.463428    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:40.486455 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.486427    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:40.488505 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.488480    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-xks-ca\") pod \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") "
Oct 31 01:48:40.488695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.488683    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-kube-api-access-twwd8\" (UniqueName: \"kubernetes.io/projected/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-kube-api-access-twwd8\") pod \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") "
Oct 31 01:48:40.488742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.488709    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-openshift-ca\") pod \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\" (UID: \"8513c054-e1f0-45c8-8f2c-1c38431ec5fc\") "
Oct 31 01:48:40.489556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.489528    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-openshift-ca" pod "8513c054-e1f0-45c8-8f2c-1c38431ec5fc" (UID: "8513c054-e1f0-45c8-8f2c-1c38431ec5fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:40.491708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.491668    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-kube-api-access-twwd8" pod "8513c054-e1f0-45c8-8f2c-1c38431ec5fc" (UID: "8513c054-e1f0-45c8-8f2c-1c38431ec5fc"). InnerVolumeSpecName "kube-api-access-twwd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:40.492457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.492439    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-xks-ca" pod "8513c054-e1f0-45c8-8f2c-1c38431ec5fc" (UID: "8513c054-e1f0-45c8-8f2c-1c38431ec5fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:48:40.648214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.647151    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twwd8\" (UniqueName: \"kubernetes.io/projected/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-kube-api-access-twwd8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:40.648214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.647184    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:40.648214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.647203    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513c054-e1f0-45c8-8f2c-1c38431ec5fc-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:40.660953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.660869    5293 patch_prober.go:29] interesting pod/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:48:40.661347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.661295    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:48:40.816107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:40.816076    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:48:41.021870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.021776    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"
Oct 31 01:48:41.021870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.021800    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb" event={"ID":"8513c054-e1f0-45c8-8f2c-1c38431ec5fc","Type":"ContainerDied","Data":"ef64089b2dba411ed7d8898e8f4abd07026d8325ee44beba723978b6effbc020"}
Oct 31 01:48:41.021870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.021859    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef64089b2dba411ed7d8898e8f4abd07026d8325ee44beba723978b6effbc020"
Oct 31 01:48:41.022420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.022303    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:41.027305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.027062    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:48:41.451073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.447160    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.495436    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-podinfo\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.495533    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-kube-api-access-fs6cr\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-kube-api-access-fs6cr\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.495609    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-token\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-token\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.495689    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-envoy\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.495727    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-certs\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.495805    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-credential-socket\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.495901    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e96e89-496f-4e62-a7c0-64773daa2b67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e96e89-496f-4e62-a7c0-64773daa2b67-istiod-ca-cert\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.495977    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-data\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.523601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.496024    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-socket\") pod \"40e96e89-496f-4e62-a7c0-64773daa2b67\" (UID: \"40e96e89-496f-4e62-a7c0-64773daa2b67\") "
Oct 31 01:48:41.525893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.525659    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-socket" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:41.526448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.526248    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-certs" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:41.530063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.526609    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-credential-socket" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:41.540964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.530341    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e96e89-496f-4e62-a7c0-64773daa2b67-istiod-ca-cert" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:41.559491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.541254    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-data" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:41.559491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.546509    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-podinfo" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:48:41.559491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.550361    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-token" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:41.573793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.573726    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-envoy" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:41.576241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.576208    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-kube-api-access-fs6cr" pod "40e96e89-496f-4e62-a7c0-64773daa2b67" (UID: "40e96e89-496f-4e62-a7c0-64773daa2b67"). InnerVolumeSpecName "kube-api-access-fs6cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627325    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs6cr\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-kube-api-access-fs6cr\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627358    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627375    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627388    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627403    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627417    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e96e89-496f-4e62-a7c0-64773daa2b67-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627431    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627444    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e96e89-496f-4e62-a7c0-64773daa2b67-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.627486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:41.627460    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e96e89-496f-4e62-a7c0-64773daa2b67-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:41.780457 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:41.780362    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:48:42.093999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.091465    5293 generic.go:356] "Generic (PLEG): container finished" podID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerID="9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29" exitCode=0
Oct 31 01:48:42.093999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.093557    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"
Oct 31 01:48:42.093999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.093804    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" event={"ID":"40e96e89-496f-4e62-a7c0-64773daa2b67","Type":"ContainerDied","Data":"9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29"}
Oct 31 01:48:42.093999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.093874    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll" event={"ID":"40e96e89-496f-4e62-a7c0-64773daa2b67","Type":"ContainerDied","Data":"80755274a612f8025f6e6bd21e572bd5b33facd8a59df358b1a88a071a4b209c"}
Oct 31 01:48:42.093999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.093905    5293 scope.go:122] "RemoveContainer" containerID="9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29"
Oct 31 01:48:42.119115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.119089    5293 scope.go:122] "RemoveContainer" containerID="fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71"
Oct 31 01:48:42.121956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.121932    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"]
Oct 31 01:48:42.127202 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.127176    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-50ac12-openshift-default-65d95897c-ljcll"]
Oct 31 01:48:42.135958 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.135937    5293 scope.go:122] "RemoveContainer" containerID="9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29"
Oct 31 01:48:42.136346 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:42.136322    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29\": container with ID starting with 9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29 not found: ID does not exist" containerID="9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29"
Oct 31 01:48:42.136421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.136357    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29"} err="failed to get container status \"9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29\": rpc error: code = NotFound desc = could not find container \"9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29\": container with ID starting with 9f7d7cdc6d982a44cbb49a1cfbdd556e25b31a4791aa91d701121a86e7e88c29 not found: ID does not exist"
Oct 31 01:48:42.136421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.136380    5293 scope.go:122] "RemoveContainer" containerID="fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71"
Oct 31 01:48:42.136706 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:42.136675    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71\": container with ID starting with fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71 not found: ID does not exist" containerID="fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71"
Oct 31 01:48:42.136772 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.136706    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71"} err="failed to get container status \"fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71\": rpc error: code = NotFound desc = could not find container \"fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71\": container with ID starting with fef6ff5ef775ce122e06aa027c67b6d1a615c0e53278b34e83ba7cc54fe70b71 not found: ID does not exist"
Oct 31 01:48:42.178015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.177983    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" path="/var/lib/kubelet/pods/40e96e89-496f-4e62-a7c0-64773daa2b67/volumes"
Oct 31 01:48:42.437741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.437709    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"]
Oct 31 01:48:42.438850 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.438824    5293 cpu_manager.go:414] "containerMap: removing container" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy"
Oct 31 01:48:42.438850 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.438851    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy"
Oct 31 01:48:42.439020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.438886    5293 cpu_manager.go:414] "containerMap: removing container" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy"
Oct 31 01:48:42.439020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.438894    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy"
Oct 31 01:48:42.439020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.438938    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8513c054-e1f0-45c8-8f2c-1c38431ec5fc" containerName="revoke-keys"
Oct 31 01:48:42.439020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.438945    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8513c054-e1f0-45c8-8f2c-1c38431ec5fc" containerName="revoke-keys"
Oct 31 01:48:42.439269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.439139    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy"
Oct 31 01:48:42.439269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.439164    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8513c054-e1f0-45c8-8f2c-1c38431ec5fc" containerName="revoke-keys"
Oct 31 01:48:42.445978 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.445958    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.451738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.451646    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-40a891-openshift-default-dockercfg-9s6sl\""
Oct 31 01:48:42.462114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.461661    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"]
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543458    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-socket\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543514    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-certs\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543597    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-token\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543677    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-podinfo\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543716    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrftf\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-kube-api-access-vrftf\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543787    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1942154-283c-4d10-acb4-4a05d07a8ac5-istiod-ca-cert\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543819    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-credential-socket\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543860    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-data\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.546588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.543948    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-envoy\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647108    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-token\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647210    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-podinfo\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647251    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrftf\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-kube-api-access-vrftf\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647315    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1942154-283c-4d10-acb4-4a05d07a8ac5-istiod-ca-cert\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647345    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-credential-socket\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647383    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-data\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647468    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-envoy\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647630    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-socket\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.647658    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-certs\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.659069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.648095    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-certs\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.695721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.692772    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-credential-socket\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.695721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.693757    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1942154-283c-4d10-acb4-4a05d07a8ac5-istiod-ca-cert\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.746823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.727263    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-podinfo\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.746823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.727884    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-data\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.746823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.728063    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-socket\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.746823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.728563    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-token\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.791081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.761798    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrftf\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-kube-api-access-vrftf\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.806731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.797659    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-envoy\") pod \"e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") " pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:42.994156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:42.993979    5293 scope.go:122] "RemoveContainer" containerID="eae40727a8a56fc7a7548f836baa74a42b68b390bc3c7a915e898f50a006dd58"
Oct 31 01:48:43.067728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.067701    5293 scope.go:122] "RemoveContainer" containerID="39624fda6f7c9418bc9741cf7abee4f44f11bc71f8b218e74b4d949dee1b37bd"
Oct 31 01:48:43.079105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.079075    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:43.080592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.080569    5293 scope.go:122] "RemoveContainer" containerID="9766ea1dc401520c40671096ac734d8f8d4b2d9f288f461bb9bc044db3ff6e49"
Oct 31 01:48:43.101326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.101304    5293 scope.go:122] "RemoveContainer" containerID="88ad6703b9b2e59556d58a25d57b46804762c736e95449ae5643922ef48e3260"
Oct 31 01:48:43.114713 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.114690    5293 scope.go:122] "RemoveContainer" containerID="1403b5aa7dc6f16bdb5a60095785923c65a0533504baca896ea8fea92f4f9575"
Oct 31 01:48:43.127545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.127497    5293 scope.go:122] "RemoveContainer" containerID="acc6718d5b7a087fd7d6f7c6e46425f7a9865e49464428a2dceb586c1f7ed95c"
Oct 31 01:48:43.161093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.159221    5293 scope.go:122] "RemoveContainer" containerID="8948b79ed9a166fe9d42386a63da192ac48f0c84391488e1815eee70f2b2db4b"
Oct 31 01:48:43.186754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.186577    5293 scope.go:122] "RemoveContainer" containerID="3361116c67eafdd05403fb7dab6c1a5183027130efd0a731cc37896f316a93ec"
Oct 31 01:48:43.397529 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:48:43.397492    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1942154_283c_4d10_acb4_4a05d07a8ac5.slice/crio-5718ff5c7c27505911e556fac20495915fe49821cc44bfa3fc9325d312a08675 WatchSource:0}: Error finding container 5718ff5c7c27505911e556fac20495915fe49821cc44bfa3fc9325d312a08675: Status 404 returned error can't find the container with id 5718ff5c7c27505911e556fac20495915fe49821cc44bfa3fc9325d312a08675
Oct 31 01:48:43.398337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.398310    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"]
Oct 31 01:48:43.400500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.400464    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:43.400585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.400542    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:43.400624 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:43.400585    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:44.135069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:44.134062    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" event={"ID":"e1942154-283c-4d10-acb4-4a05d07a8ac5","Type":"ContainerStarted","Data":"cbb4796c756c3ad3ad41b6a2c3a4cb12604810c479c2deed9e5cf88f141c3edc"}
Oct 31 01:48:44.135069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:44.134128    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" event={"ID":"e1942154-283c-4d10-acb4-4a05d07a8ac5","Type":"ContainerStarted","Data":"5718ff5c7c27505911e556fac20495915fe49821cc44bfa3fc9325d312a08675"}
Oct 31 01:48:44.195966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:44.194360    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podStartSLOduration=2.194337578 podStartE2EDuration="2.194337578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:44.172397129 +0000 UTC m=+1682.121045526" watchObservedRunningTime="2026-10-31 01:48:44.194337578 +0000 UTC m=+1682.142985971"
Oct 31 01:48:44.515892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:44.514597    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": read tcp 10.133.0.2:44510->10.133.0.136:15021: read: connection reset by peer" start-of-body=
Oct 31 01:48:44.515892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:44.514652    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": read tcp 10.133.0.2:44510->10.133.0.136:15021: read: connection reset by peer"
Oct 31 01:48:45.079800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.079717    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:45.085278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.085252    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:45.145405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.145371    5293 generic.go:356] "Generic (PLEG): container finished" podID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerID="2e1cbff65f4405c2a8cecd8ac5bb497a488abb505635869b31a7335ac22bf817" exitCode=0
Oct 31 01:48:45.145803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.145444    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" event={"ID":"546bcf87-5caf-4136-8a1f-d6719ba577cb","Type":"ContainerDied","Data":"2e1cbff65f4405c2a8cecd8ac5bb497a488abb505635869b31a7335ac22bf817"}
Oct 31 01:48:45.145892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.145831    5293 scope.go:122] "RemoveContainer" containerID="2e1cbff65f4405c2a8cecd8ac5bb497a488abb505635869b31a7335ac22bf817"
Oct 31 01:48:45.145961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.145915    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:45.146822 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.146776    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:48:45.147337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.147309    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:45.147398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.147376    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:45.147436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:45.147405    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:46.155945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:46.155911    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" event={"ID":"546bcf87-5caf-4136-8a1f-d6719ba577cb","Type":"ContainerStarted","Data":"78d2efcf868ff1117db4c3cd0ee38f56b3e28ce7d57a136f5bc003f2254bb0de"}
Oct 31 01:48:46.555775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:46.555682    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:48:46.555951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:46.555870    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 01:48:46.555951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:46.555920    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 01:48:46.794382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:46.794346    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"]
Oct 31 01:48:46.842465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:46.842361    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"]
Oct 31 01:48:47.555541 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:47.555505    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 01:48:47.556171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:47.556134    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 01:48:48.058435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.001332    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"]
Oct 31 01:48:48.058435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.002544    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e96e89-496f-4e62-a7c0-64773daa2b67" containerName="istio-proxy"
Oct 31 01:48:48.058435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.050484    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:48.084082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.068763    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-40a891-dockercfg-4hctv\""
Oct 31 01:48:48.123921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.084618    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-40a891\""
Oct 31 01:48:48.123921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.096658    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d984d62-3baf-4860-8f46-d5f3849ac4cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9\" (UID: \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:48.123921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.096850    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwtz\" (UniqueName: \"kubernetes.io/projected/5d984d62-3baf-4860-8f46-d5f3849ac4cd-kube-api-access-4pwtz\") pod \"payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9\" (UID: \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:48.247064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.209314    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d984d62-3baf-4860-8f46-d5f3849ac4cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9\" (UID: \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:48.542120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.264741    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d984d62-3baf-4860-8f46-d5f3849ac4cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9\" (UID: \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:48.542120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.314958    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwtz\" (UniqueName: \"kubernetes.io/projected/5d984d62-3baf-4860-8f46-d5f3849ac4cd-kube-api-access-4pwtz\") pod \"payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9\" (UID: \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:48.561393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.561361    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"]
Oct 31 01:48:48.601398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.601366    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:48.636102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.632890    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 01:48:48.636102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.632958    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 01:48:48.636102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.635570    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"]
Oct 31 01:48:48.659105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.644991    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwtz\" (UniqueName: \"kubernetes.io/projected/5d984d62-3baf-4860-8f46-d5f3849ac4cd-kube-api-access-4pwtz\") pod \"payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9\" (UID: \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:48.677796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.677760    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrpr\" (UniqueName: \"kubernetes.io/projected/8211cfe6-7aee-43b9-b5f5-439ffb348931-kube-api-access-4zrpr\") pod \"payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg\" (UID: \"8211cfe6-7aee-43b9-b5f5-439ffb348931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:48.678258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.678232    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8211cfe6-7aee-43b9-b5f5-439ffb348931-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg\" (UID: \"8211cfe6-7aee-43b9-b5f5-439ffb348931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:48.806361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.777565    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"]
Oct 31 01:48:48.834082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.791421    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8211cfe6-7aee-43b9-b5f5-439ffb348931-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg\" (UID: \"8211cfe6-7aee-43b9-b5f5-439ffb348931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:48.834082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.809649    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrpr\" (UniqueName: \"kubernetes.io/projected/8211cfe6-7aee-43b9-b5f5-439ffb348931-kube-api-access-4zrpr\") pod \"payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg\" (UID: \"8211cfe6-7aee-43b9-b5f5-439ffb348931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:48.834082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.792526    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8211cfe6-7aee-43b9-b5f5-439ffb348931-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg\" (UID: \"8211cfe6-7aee-43b9-b5f5-439ffb348931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:48.868087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.865731    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrpr\" (UniqueName: \"kubernetes.io/projected/8211cfe6-7aee-43b9-b5f5-439ffb348931-kube-api-access-4zrpr\") pod \"payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg\" (UID: \"8211cfe6-7aee-43b9-b5f5-439ffb348931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:48.883401 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:48.881782    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:49.078343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:49.077615    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:49.352775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:49.352675    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"]
Oct 31 01:48:49.644122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:49.623108    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 01:48:49.644122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:49.623168    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 01:48:49.786587 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:49.786405    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9" event={"ID":"5d984d62-3baf-4860-8f46-d5f3849ac4cd","Type":"ContainerStarted","Data":"4c969d03f60330c7e85016680a019399dd0e9fba188a44cc84aa460291a32ff7"}
Oct 31 01:48:50.036248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.015813    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"]
Oct 31 01:48:50.175540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.166947    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"]
Oct 31 01:48:50.273631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.273600    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703b524-bhhpr"]
Oct 31 01:48:50.561166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.560358    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 01:48:50.561166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.560415    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 01:48:50.827122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.826882    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg" event={"ID":"8211cfe6-7aee-43b9-b5f5-439ffb348931","Type":"ContainerStarted","Data":"9eb6257b70fda2c3597230217ed844e85b6a44203ad7528d6c2317dd08231972"}
Oct 31 01:48:50.827122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.826956    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg" event={"ID":"8211cfe6-7aee-43b9-b5f5-439ffb348931","Type":"ContainerStarted","Data":"e49ef85da29a7a44bcea2ed6e827eef5fb8746013ad847328f9b7e2964d96346"}
Oct 31 01:48:50.854123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.828249    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:48:50.948066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.943237    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg" podStartSLOduration=3.943216909 podStartE2EDuration="3.943216909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:50.941713992 +0000 UTC m=+1688.890362400" watchObservedRunningTime="2026-10-31 01:48:50.943216909 +0000 UTC m=+1688.891865302"
Oct 31 01:48:50.969064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.950301    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9" event={"ID":"5d984d62-3baf-4860-8f46-d5f3849ac4cd","Type":"ContainerStarted","Data":"f73ecae6d9b76dca08a0919c26856bb4eefbef76a27f20d4c207efdbd37b70bd"}
Oct 31 01:48:50.969064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:50.951510    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:48:51.061085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:51.047911    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9" podStartSLOduration=4.047889849 podStartE2EDuration="4.047889849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:51.018931515 +0000 UTC m=+1688.967579910" watchObservedRunningTime="2026-10-31 01:48:51.047889849 +0000 UTC m=+1688.996538242"
Oct 31 01:48:51.560094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:51.559491    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 01:48:51.560094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:51.559563    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 01:48:51.796550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:51.796343    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg" podUID="f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3" containerName="payload-processing" containerID="cri-o://3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636" gracePeriod=25
Oct 31 01:48:51.844764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:51.844102    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv" podUID="f25d9e39-2355-4ca5-80cb-dbeebf95fc26" containerName="payload-pre-processing" containerID="cri-o://dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479" gracePeriod=25
Oct 31 01:48:52.185445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.185405    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13aa1f63-03a8-425b-8385-9177f86d95b7" path="/var/lib/kubelet/pods/13aa1f63-03a8-425b-8385-9177f86d95b7/volumes"
Oct 31 01:48:52.761257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.760135    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:48:52.763316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.762200    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:48:52.862253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.856195    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:48:52.862253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.857973    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:48:52.862253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.860405    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:48:52.872207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.871893    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-kube-api-access-gwskn\" (UniqueName: \"kubernetes.io/projected/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-kube-api-access-gwskn\") pod \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\" (UID: \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\") "
Oct 31 01:48:52.872207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.871960    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-plugins-config-volume\") pod \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\" (UID: \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\") "
Oct 31 01:48:52.872207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.872004    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-kube-api-access-686rg\" (UniqueName: \"kubernetes.io/projected/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-kube-api-access-686rg\") pod \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\" (UID: \"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3\") "
Oct 31 01:48:52.872503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.872460    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-plugins-config-volume\") pod \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\" (UID: \"f25d9e39-2355-4ca5-80cb-dbeebf95fc26\") "
Oct 31 01:48:52.875964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.875751    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-plugins-config-volume" pod "f25d9e39-2355-4ca5-80cb-dbeebf95fc26" (UID: "f25d9e39-2355-4ca5-80cb-dbeebf95fc26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:52.882402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.882363    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-kube-api-access-gwskn" pod "f25d9e39-2355-4ca5-80cb-dbeebf95fc26" (UID: "f25d9e39-2355-4ca5-80cb-dbeebf95fc26"). InnerVolumeSpecName "kube-api-access-gwskn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:52.882780 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.882754    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-plugins-config-volume" pod "f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3" (UID: "f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:52.888339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.888310    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-kube-api-access-686rg" pod "f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3" (UID: "f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3"). InnerVolumeSpecName "kube-api-access-686rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:52.976962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.976710    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:52.976962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.976741    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwskn\" (UniqueName: \"kubernetes.io/projected/f25d9e39-2355-4ca5-80cb-dbeebf95fc26-kube-api-access-gwskn\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:52.976962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.976786    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:52.976962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:52.976802    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686rg\" (UniqueName: \"kubernetes.io/projected/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3-kube-api-access-686rg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:53.039127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.033720    5293 generic.go:356] "Generic (PLEG): container finished" podID="f25d9e39-2355-4ca5-80cb-dbeebf95fc26" containerID="dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479" exitCode=0
Oct 31 01:48:53.040152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.039522    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv" event={"ID":"f25d9e39-2355-4ca5-80cb-dbeebf95fc26","Type":"ContainerDied","Data":"dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479"}
Oct 31 01:48:53.040152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.039686    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv" event={"ID":"f25d9e39-2355-4ca5-80cb-dbeebf95fc26","Type":"ContainerDied","Data":"4d507ef0e3c44c5c3ab6efcb7aae6cbefabd024d718fd4b5301e34e5d51c948f"}
Oct 31 01:48:53.040152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.039745    5293 scope.go:122] "RemoveContainer" containerID="dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479"
Oct 31 01:48:53.040700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.040609    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"
Oct 31 01:48:53.042692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.042669    5293 generic.go:356] "Generic (PLEG): container finished" podID="f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3" containerID="3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636" exitCode=0
Oct 31 01:48:53.044327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.044298    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"
Oct 31 01:48:53.044601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.044574    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg" event={"ID":"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3","Type":"ContainerDied","Data":"3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636"}
Oct 31 01:48:53.044688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.044621    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg" event={"ID":"f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3","Type":"ContainerDied","Data":"3397ef1e66131e919ae2c40b38c39345f99ff2109abbd79ffacb84f54c803dff"}
Oct 31 01:48:53.056303 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.056283    5293 scope.go:122] "RemoveContainer" containerID="dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479"
Oct 31 01:48:53.056740 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:53.056704    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479\": container with ID starting with dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479 not found: ID does not exist" containerID="dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479"
Oct 31 01:48:53.056808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.056777    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479"} err="failed to get container status \"dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479\": rpc error: code = NotFound desc = could not find container \"dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479\": container with ID starting with dccaf386afdd2348c2ae1aa52dfe20b1608f69f342b3e79a969957e638808479 not found: ID does not exist"
Oct 31 01:48:53.056808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.056796    5293 scope.go:122] "RemoveContainer" containerID="3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636"
Oct 31 01:48:53.197135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.195300    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"]
Oct 31 01:48:53.228243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.228209    5293 scope.go:122] "RemoveContainer" containerID="3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636"
Oct 31 01:48:53.229011 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:48:53.228894    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636\": container with ID starting with 3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636 not found: ID does not exist" containerID="3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636"
Oct 31 01:48:53.229011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.228931    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636"} err="failed to get container status \"3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636\": rpc error: code = NotFound desc = could not find container \"3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636\": container with ID starting with 3560e1ec5b07bd8fdc5f276092887bd44d63cb7487a96373221ecda51a444636 not found: ID does not exist"
Oct 31 01:48:53.229011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.228963    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea6cc6-66f9dc7bb7-8fhxv"]
Oct 31 01:48:53.243071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.237117    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"]
Oct 31 01:48:53.243242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.243119    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea6cc6-575b4694c7-29mgg"]
Oct 31 01:48:53.656035 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.656006    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"]
Oct 31 01:48:53.660009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.659986    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3" containerName="payload-processing"
Oct 31 01:48:53.660205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.660189    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3" containerName="payload-processing"
Oct 31 01:48:53.660366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.660351    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f25d9e39-2355-4ca5-80cb-dbeebf95fc26" containerName="payload-pre-processing"
Oct 31 01:48:53.660467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.660454    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25d9e39-2355-4ca5-80cb-dbeebf95fc26" containerName="payload-pre-processing"
Oct 31 01:48:53.662245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.662224    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25d9e39-2355-4ca5-80cb-dbeebf95fc26" containerName="payload-pre-processing"
Oct 31 01:48:53.662390 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.662377    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3" containerName="payload-processing"
Oct 31 01:48:53.675253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.675234    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"]
Oct 31 01:48:53.675431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.675418    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"]
Oct 31 01:48:53.679471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.679453    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:53.682916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.682899    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"]
Oct 31 01:48:53.683148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.683132    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:53.688162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.687398    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cbbaec\""
Oct 31 01:48:53.688162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.688030    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cbbaec-dockercfg-9cvpl\""
Oct 31 01:48:53.791176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.791134    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzc5\" (UniqueName: \"kubernetes.io/projected/677350f3-0c22-4e94-a120-870eec7a20b5-kube-api-access-8kzc5\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr\" (UID: \"677350f3-0c22-4e94-a120-870eec7a20b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:53.791374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.791213    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677350f3-0c22-4e94-a120-870eec7a20b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr\" (UID: \"677350f3-0c22-4e94-a120-870eec7a20b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:53.791374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.791245    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d743ff0-4856-42bf-804b-ff2f686b2100-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd\" (UID: \"2d743ff0-4856-42bf-804b-ff2f686b2100\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:53.791374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.791272    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jrt7\" (UniqueName: \"kubernetes.io/projected/2d743ff0-4856-42bf-804b-ff2f686b2100-kube-api-access-7jrt7\") pod \"payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd\" (UID: \"2d743ff0-4856-42bf-804b-ff2f686b2100\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:53.916659 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.916563    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677350f3-0c22-4e94-a120-870eec7a20b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr\" (UID: \"677350f3-0c22-4e94-a120-870eec7a20b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:53.950869 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.924809    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677350f3-0c22-4e94-a120-870eec7a20b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr\" (UID: \"677350f3-0c22-4e94-a120-870eec7a20b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:53.955814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.954164    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d743ff0-4856-42bf-804b-ff2f686b2100-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd\" (UID: \"2d743ff0-4856-42bf-804b-ff2f686b2100\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:53.965446 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.951461    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d743ff0-4856-42bf-804b-ff2f686b2100-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd\" (UID: \"2d743ff0-4856-42bf-804b-ff2f686b2100\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:53.977784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:53.977746    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jrt7\" (UniqueName: \"kubernetes.io/projected/2d743ff0-4856-42bf-804b-ff2f686b2100-kube-api-access-7jrt7\") pod \"payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd\" (UID: \"2d743ff0-4856-42bf-804b-ff2f686b2100\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:54.001766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.001203    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzc5\" (UniqueName: \"kubernetes.io/projected/677350f3-0c22-4e94-a120-870eec7a20b5-kube-api-access-8kzc5\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr\" (UID: \"677350f3-0c22-4e94-a120-870eec7a20b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:54.191900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.179336    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jrt7\" (UniqueName: \"kubernetes.io/projected/2d743ff0-4856-42bf-804b-ff2f686b2100-kube-api-access-7jrt7\") pod \"payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd\" (UID: \"2d743ff0-4856-42bf-804b-ff2f686b2100\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:54.191900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.179902    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzc5\" (UniqueName: \"kubernetes.io/projected/677350f3-0c22-4e94-a120-870eec7a20b5-kube-api-access-8kzc5\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr\" (UID: \"677350f3-0c22-4e94-a120-870eec7a20b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:54.191900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.188371    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:54.287205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.287177    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:54.346064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.323091    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25d9e39-2355-4ca5-80cb-dbeebf95fc26" path="/var/lib/kubelet/pods/f25d9e39-2355-4ca5-80cb-dbeebf95fc26/volumes"
Oct 31 01:48:54.369968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.369937    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3" path="/var/lib/kubelet/pods/f69cd0c8-578c-48ba-89e3-f3a6cc64c9d3/volumes"
Oct 31 01:48:54.720471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.718261    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"]
Oct 31 01:48:54.724751 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:48:54.724615    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d743ff0_4856_42bf_804b_ff2f686b2100.slice/crio-f64b6b040f646a07341b219794d68e1de72e0bd3c82ba87208210e9469c817bb WatchSource:0}: Error finding container f64b6b040f646a07341b219794d68e1de72e0bd3c82ba87208210e9469c817bb: Status 404 returned error can't find the container with id f64b6b040f646a07341b219794d68e1de72e0bd3c82ba87208210e9469c817bb
Oct 31 01:48:54.726807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:54.726787    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"]
Oct 31 01:48:54.727158 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:48:54.727136    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677350f3_0c22_4e94_a120_870eec7a20b5.slice/crio-915a4d3078c86d8ca0b9892f99f57b21926a597043fe1905d01c4440566142fd WatchSource:0}: Error finding container 915a4d3078c86d8ca0b9892f99f57b21926a597043fe1905d01c4440566142fd: Status 404 returned error can't find the container with id 915a4d3078c86d8ca0b9892f99f57b21926a597043fe1905d01c4440566142fd
Oct 31 01:48:55.387611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:55.387578    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd" event={"ID":"2d743ff0-4856-42bf-804b-ff2f686b2100","Type":"ContainerStarted","Data":"d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949"}
Oct 31 01:48:55.388113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:55.387673    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd" event={"ID":"2d743ff0-4856-42bf-804b-ff2f686b2100","Type":"ContainerStarted","Data":"f64b6b040f646a07341b219794d68e1de72e0bd3c82ba87208210e9469c817bb"}
Oct 31 01:48:55.388113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:55.387708    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:48:55.389427 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:55.389396    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr" event={"ID":"677350f3-0c22-4e94-a120-870eec7a20b5","Type":"ContainerStarted","Data":"48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1"}
Oct 31 01:48:55.389512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:55.389438    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr" event={"ID":"677350f3-0c22-4e94-a120-870eec7a20b5","Type":"ContainerStarted","Data":"915a4d3078c86d8ca0b9892f99f57b21926a597043fe1905d01c4440566142fd"}
Oct 31 01:48:55.389563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:55.389519    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:48:55.409465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:55.409413    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd" podStartSLOduration=2.409395262 podStartE2EDuration="2.409395262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:55.404559312 +0000 UTC m=+1693.353207705" watchObservedRunningTime="2026-10-31 01:48:55.409395262 +0000 UTC m=+1693.358043655"
Oct 31 01:48:55.432279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:55.432227    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr" podStartSLOduration=2.432209183 podStartE2EDuration="2.432209183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:55.421466747 +0000 UTC m=+1693.370115140" watchObservedRunningTime="2026-10-31 01:48:55.432209183 +0000 UTC m=+1693.380857577"
Oct 31 01:48:57.969064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:57.968140    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"]
Oct 31 01:48:58.016002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:58.002222    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea6cc6-zlpxb"]
Oct 31 01:48:58.205199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:58.205158    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8513c054-e1f0-45c8-8f2c-1c38431ec5fc" path="/var/lib/kubelet/pods/8513c054-e1f0-45c8-8f2c-1c38431ec5fc/volumes"
Oct 31 01:48:59.531170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:59.528821    5293 generic.go:356] "Generic (PLEG): container finished" podID="ad61337c-517a-4100-894d-d0169a6e69a1" containerID="8cd4cdb95c2c7f30dd330fae4f2860e9052663fc3de469a7f54ddc7e6acf4e63" exitCode=0
Oct 31 01:48:59.531170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:59.528975    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" event={"ID":"ad61337c-517a-4100-894d-d0169a6e69a1","Type":"ContainerDied","Data":"8cd4cdb95c2c7f30dd330fae4f2860e9052663fc3de469a7f54ddc7e6acf4e63"}
Oct 31 01:48:59.531170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:59.529603    5293 scope.go:122] "RemoveContainer" containerID="8cd4cdb95c2c7f30dd330fae4f2860e9052663fc3de469a7f54ddc7e6acf4e63"
Oct 31 01:48:59.534721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:59.533423    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:59.534721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:59.533522    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:48:59.534721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:48:59.533574    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:00.550142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:00.550100    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" event={"ID":"ad61337c-517a-4100-894d-d0169a6e69a1","Type":"ContainerStarted","Data":"7a6487c3b79e99b91f6e5625be6f8fe347122af2e99ddf2ac2eea9f387db2832"}
Oct 31 01:49:00.870209 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:00.870111    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:49:00.871344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:00.871316    5293 patch_prober.go:29] interesting pod/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:49:00.871502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:00.871367    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:49:01.871119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:01.871018    5293 patch_prober.go:29] interesting pod/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:49:01.871119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:01.871120    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:49:02.032658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:02.032622    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"]
Oct 31 01:49:02.036708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:02.036676    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35be93c2-92s44"]
Oct 31 01:49:02.186434 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:02.186398    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db94e412-6e99-4bc8-bf33-2b258cfbaeba" path="/var/lib/kubelet/pods/db94e412-6e99-4bc8-bf33-2b258cfbaeba/volumes"
Oct 31 01:49:02.869712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:02.869679    5293 patch_prober.go:29] interesting pod/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:49:02.869894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:02.869736    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:49:03.032824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:03.032787    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"]
Oct 31 01:49:03.038893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:03.038860    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac46e55-97rlv"]
Oct 31 01:49:03.047690 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:03.047666    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:49:03.049743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:03.049724    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:49:03.583840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:03.583807    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:49:03.869567 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:03.869477    5293 patch_prober.go:29] interesting pod/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:49:03.869567 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:03.869530    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:49:04.185553 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:04.185516    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="464b1fc8-03b2-4d95-aece-0e33f134b266" path="/var/lib/kubelet/pods/464b1fc8-03b2-4d95-aece-0e33f134b266/volumes"
Oct 31 01:49:04.871518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:04.871473    5293 patch_prober.go:29] interesting pod/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:49:04.871814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:04.871783    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:49:05.873230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:05.873169    5293 patch_prober.go:29] interesting pod/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:49:05.875450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:05.875405    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:49:06.033555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:06.033521    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"]
Oct 31 01:49:06.033850 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:06.033802    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy" containerID="cri-o://7a6487c3b79e99b91f6e5625be6f8fe347122af2e99ddf2ac2eea9f387db2832" gracePeriod=30
Oct 31 01:49:06.455699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:06.455672    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:49:06.492263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:06.492230    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:49:06.676358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:06.676330    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"]
Oct 31 01:49:06.682103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:06.682070    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"]
Oct 31 01:49:07.489137 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.489099    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"]
Oct 31 01:49:07.495981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.495955    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.502692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.502666    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-073e24-openshift-default-dockercfg-qzxns\""
Oct 31 01:49:07.515344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.514787    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"]
Oct 31 01:49:07.555945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.555907    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-data\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.556174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.555956    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-istio-token\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.556174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.555983    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-certs\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.556174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.556012    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-credential-socket\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.556364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.556196    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a96b284-a772-4d38-aac9-665181786781-istio-podinfo\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.556364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.556222    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz662\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-kube-api-access-fz662\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.556364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.556252    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a96b284-a772-4d38-aac9-665181786781-istiod-ca-cert\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.556364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.556284    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-envoy\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.556364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.556300    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-socket\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662287    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a96b284-a772-4d38-aac9-665181786781-istio-podinfo\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662355    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz662\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-kube-api-access-fz662\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662417    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a96b284-a772-4d38-aac9-665181786781-istiod-ca-cert\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662478    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-envoy\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662505    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-socket\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662619    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-data\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662656    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-istio-token\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662685    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-certs\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.662726    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-credential-socket\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.663293    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-socket\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.663305    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-credential-socket\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.663521    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-data\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.666077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.663951    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-certs\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.684967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.681612    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a96b284-a772-4d38-aac9-665181786781-istiod-ca-cert\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.709078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.697398    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-envoy\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.709078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.697755    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a96b284-a772-4d38-aac9-665181786781-istio-podinfo\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.734921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.734885    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz662\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-kube-api-access-fz662\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.761685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.761613    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-istio-token\") pod \"e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") " pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:07.853263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:07.853237    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:08.311063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:08.304850    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"]
Oct 31 01:49:08.383070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:08.370920    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:08.383423 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:08.383384    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:08.383599 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:08.383581    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:08.853556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:08.853519    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" event={"ID":"5a96b284-a772-4d38-aac9-665181786781","Type":"ContainerStarted","Data":"80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d"}
Oct 31 01:49:08.854155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:08.854130    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" event={"ID":"5a96b284-a772-4d38-aac9-665181786781","Type":"ContainerStarted","Data":"8530a12c28ad2356ce250ffd0f174e05db717c2fa126ae0a8c8f8d288948aaac"}
Oct 31 01:49:08.854234 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:08.854176    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:08.879224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:08.879181    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podStartSLOduration=1.879164368 podStartE2EDuration="1.879164368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:08.875789916 +0000 UTC m=+1706.824438309" watchObservedRunningTime="2026-10-31 01:49:08.879164368 +0000 UTC m=+1706.827812762"
Oct 31 01:49:09.861427 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:09.861398    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:09.863999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:09.863979    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:09.864888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:09.864870    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:09.949939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:09.949907    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"]
Oct 31 01:49:09.957966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:09.957943    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:09.972910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:09.972878    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-40a891-dockercfg-pwtzt\""
Oct 31 01:49:10.008795 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.008762    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"]
Oct 31 01:49:10.016467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.016421    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"]
Oct 31 01:49:10.022500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.022474    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:10.030353 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.030322    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"]
Oct 31 01:49:10.044515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.044482    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdtg\" (UniqueName: \"kubernetes.io/projected/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-kube-api-access-jwdtg\") pod \"payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4\" (UID: \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:10.044685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.044668    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4\" (UID: \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:10.245064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.240505    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4\" (UID: \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:10.245064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.240712    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvb5b\" (UniqueName: \"kubernetes.io/projected/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-kube-api-access-fvb5b\") pod \"payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82\" (UID: \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:10.245064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.240831    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdtg\" (UniqueName: \"kubernetes.io/projected/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-kube-api-access-jwdtg\") pod \"payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4\" (UID: \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:10.245064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.240999    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82\" (UID: \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:10.261072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.246000    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4\" (UID: \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:10.312306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.283980    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdtg\" (UniqueName: \"kubernetes.io/projected/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-kube-api-access-jwdtg\") pod \"payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4\" (UID: \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:10.312306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.309070    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:10.392214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.372558    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82\" (UID: \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:10.392214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.372787    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvb5b\" (UniqueName: \"kubernetes.io/projected/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-kube-api-access-fvb5b\") pod \"payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82\" (UID: \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:10.392214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.385678    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82\" (UID: \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:10.456255 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.447980    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvb5b\" (UniqueName: \"kubernetes.io/projected/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-kube-api-access-fvb5b\") pod \"payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82\" (UID: \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:10.674120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.670326    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:10.753236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.753210    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"]
Oct 31 01:49:10.753973 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:49:10.753944    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dedf1a_4f23_403e_b2ce_0d1d1732884a.slice/crio-628e4cf0d7149a071551ef594f03a573c7331a5ed4ad0aac8d58997c20c7c069 WatchSource:0}: Error finding container 628e4cf0d7149a071551ef594f03a573c7331a5ed4ad0aac8d58997c20c7c069: Status 404 returned error can't find the container with id 628e4cf0d7149a071551ef594f03a573c7331a5ed4ad0aac8d58997c20c7c069
Oct 31 01:49:10.873748 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.873709    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4" event={"ID":"a6dedf1a-4f23-403e-b2ce-0d1d1732884a","Type":"ContainerStarted","Data":"2cf74b8e731308d41198e136adc525b8a1be943a7436d7bc9ee826ea2544af62"}
Oct 31 01:49:10.874262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.873773    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4" event={"ID":"a6dedf1a-4f23-403e-b2ce-0d1d1732884a","Type":"ContainerStarted","Data":"628e4cf0d7149a071551ef594f03a573c7331a5ed4ad0aac8d58997c20c7c069"}
Oct 31 01:49:10.874262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.873979    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:10.891716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.891661    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4" podStartSLOduration=1.891643371 podStartE2EDuration="1.891643371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:10.889682008 +0000 UTC m=+1708.838330401" watchObservedRunningTime="2026-10-31 01:49:10.891643371 +0000 UTC m=+1708.840291764"
Oct 31 01:49:10.977660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:10.977621    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"]
Oct 31 01:49:10.979385 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:49:10.979357    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb20b7d_f1b3_4773_be3b_335f1271d3a6.slice/crio-bdd9e929ac023fbbbf63cb9d01788e9ca76959ba576de269ff58adf147206558 WatchSource:0}: Error finding container bdd9e929ac023fbbbf63cb9d01788e9ca76959ba576de269ff58adf147206558: Status 404 returned error can't find the container with id bdd9e929ac023fbbbf63cb9d01788e9ca76959ba576de269ff58adf147206558
Oct 31 01:49:11.677703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.677667    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg" podUID="8211cfe6-7aee-43b9-b5f5-439ffb348931" containerName="payload-processing" containerID="cri-o://9eb6257b70fda2c3597230217ed844e85b6a44203ad7528d6c2317dd08231972" gracePeriod=25
Oct 31 01:49:11.683382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.683199    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9" podUID="5d984d62-3baf-4860-8f46-d5f3849ac4cd" containerName="payload-pre-processing" containerID="cri-o://f73ecae6d9b76dca08a0919c26856bb4eefbef76a27f20d4c207efdbd37b70bd" gracePeriod=25
Oct 31 01:49:11.794560 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:49:11.794526    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:49:11.886060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.885977    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82" event={"ID":"8eb20b7d-f1b3-4773-be3b-335f1271d3a6","Type":"ContainerStarted","Data":"11cbcaeebf2435f992f1d773df32b4df964527421291d02652a9809a361d368a"}
Oct 31 01:49:11.886060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.886051    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82" event={"ID":"8eb20b7d-f1b3-4773-be3b-335f1271d3a6","Type":"ContainerStarted","Data":"bdd9e929ac023fbbbf63cb9d01788e9ca76959ba576de269ff58adf147206558"}
Oct 31 01:49:11.886614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.886302    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:11.888809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.888528    5293 generic.go:356] "Generic (PLEG): container finished" podID="8211cfe6-7aee-43b9-b5f5-439ffb348931" containerID="9eb6257b70fda2c3597230217ed844e85b6a44203ad7528d6c2317dd08231972" exitCode=0
Oct 31 01:49:11.888809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.888710    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg" event={"ID":"8211cfe6-7aee-43b9-b5f5-439ffb348931","Type":"ContainerDied","Data":"9eb6257b70fda2c3597230217ed844e85b6a44203ad7528d6c2317dd08231972"}
Oct 31 01:49:11.891829 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.891684    5293 generic.go:356] "Generic (PLEG): container finished" podID="5d984d62-3baf-4860-8f46-d5f3849ac4cd" containerID="f73ecae6d9b76dca08a0919c26856bb4eefbef76a27f20d4c207efdbd37b70bd" exitCode=0
Oct 31 01:49:11.891829 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.891782    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9" event={"ID":"5d984d62-3baf-4860-8f46-d5f3849ac4cd","Type":"ContainerDied","Data":"f73ecae6d9b76dca08a0919c26856bb4eefbef76a27f20d4c207efdbd37b70bd"}
Oct 31 01:49:11.893676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.893661    5293 generic.go:356] "Generic (PLEG): container finished" podID="ad61337c-517a-4100-894d-d0169a6e69a1" containerID="7a6487c3b79e99b91f6e5625be6f8fe347122af2e99ddf2ac2eea9f387db2832" exitCode=0
Oct 31 01:49:11.901107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.893794    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" event={"ID":"ad61337c-517a-4100-894d-d0169a6e69a1","Type":"ContainerDied","Data":"7a6487c3b79e99b91f6e5625be6f8fe347122af2e99ddf2ac2eea9f387db2832"}
Oct 31 01:49:11.901107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.900291    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt" event={"ID":"ad61337c-517a-4100-894d-d0169a6e69a1","Type":"ContainerDied","Data":"f84792b52af38c2f2da2709496fcab10674648fa0c55a05c5dc5382c67ca4e1f"}
Oct 31 01:49:11.901107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.900322    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f84792b52af38c2f2da2709496fcab10674648fa0c55a05c5dc5382c67ca4e1f"
Oct 31 01:49:11.901107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.900340    5293 scope.go:122] "RemoveContainer" containerID="8cd4cdb95c2c7f30dd330fae4f2860e9052663fc3de469a7f54ddc7e6acf4e63"
Oct 31 01:49:11.912877 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.912566    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:49:11.927541 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:11.915877    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82" podStartSLOduration=2.915861578 podStartE2EDuration="2.915861578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:11.910672186 +0000 UTC m=+1709.859320580" watchObservedRunningTime="2026-10-31 01:49:11.915861578 +0000 UTC m=+1709.864509971"
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.023582    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad61337c-517a-4100-894d-d0169a6e69a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61337c-517a-4100-894d-d0169a6e69a1-istio-podinfo\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.023648    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-envoy\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.023794    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-socket\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.023894    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-credential-socket\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.023991    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad61337c-517a-4100-894d-d0169a6e69a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61337c-517a-4100-894d-d0169a6e69a1-istiod-ca-cert\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.024060    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-certs\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.024095    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-kube-api-access-2mfg9\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-kube-api-access-2mfg9\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.024174    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-istio-token\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-istio-token\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.024266    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-data\") pod \"ad61337c-517a-4100-894d-d0169a6e69a1\" (UID: \"ad61337c-517a-4100-894d-d0169a6e69a1\") "
Oct 31 01:49:12.025927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.025846    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-data" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:12.029253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.029188    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-socket" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:12.030704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.030656    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-certs" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:12.031495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.031255    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad61337c-517a-4100-894d-d0169a6e69a1-istiod-ca-cert" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:12.041227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.035631    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-credential-socket" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:12.041227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.036096    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-kube-api-access-2mfg9" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "kube-api-access-2mfg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:12.041227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.037230    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-istio-token" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:12.051932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.051895    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad61337c-517a-4100-894d-d0169a6e69a1-istio-podinfo" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:49:12.053914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.053888    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-envoy" pod "ad61337c-517a-4100-894d-d0169a6e69a1" (UID: "ad61337c-517a-4100-894d-d0169a6e69a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:12.054030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.053956    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:49:12.083184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.083160    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.126683    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8211cfe6-7aee-43b9-b5f5-439ffb348931-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8211cfe6-7aee-43b9-b5f5-439ffb348931-plugins-config-volume\") pod \"8211cfe6-7aee-43b9-b5f5-439ffb348931\" (UID: \"8211cfe6-7aee-43b9-b5f5-439ffb348931\") "
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.137163    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8211cfe6-7aee-43b9-b5f5-439ffb348931-kube-api-access-4zrpr\" (UniqueName: \"kubernetes.io/projected/8211cfe6-7aee-43b9-b5f5-439ffb348931-kube-api-access-4zrpr\") pod \"8211cfe6-7aee-43b9-b5f5-439ffb348931\" (UID: \"8211cfe6-7aee-43b9-b5f5-439ffb348931\") "
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.137467    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d984d62-3baf-4860-8f46-d5f3849ac4cd-kube-api-access-4pwtz\" (UniqueName: \"kubernetes.io/projected/5d984d62-3baf-4860-8f46-d5f3849ac4cd-kube-api-access-4pwtz\") pod \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\" (UID: \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\") "
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.137570    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d984d62-3baf-4860-8f46-d5f3849ac4cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d984d62-3baf-4860-8f46-d5f3849ac4cd-plugins-config-volume\") pod \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\" (UID: \"5d984d62-3baf-4860-8f46-d5f3849ac4cd\") "
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138350    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61337c-517a-4100-894d-d0169a6e69a1-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138372    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138390    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138404    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138417    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61337c-517a-4100-894d-d0169a6e69a1-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138433    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138447    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfg9\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-kube-api-access-2mfg9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138461    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad61337c-517a-4100-894d-d0169a6e69a1-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.138473    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61337c-517a-4100-894d-d0169a6e69a1-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.139070    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8211cfe6-7aee-43b9-b5f5-439ffb348931-plugins-config-volume" pod "8211cfe6-7aee-43b9-b5f5-439ffb348931" (UID: "8211cfe6-7aee-43b9-b5f5-439ffb348931"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:12.144112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.140015    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d984d62-3baf-4860-8f46-d5f3849ac4cd-plugins-config-volume" pod "5d984d62-3baf-4860-8f46-d5f3849ac4cd" (UID: "5d984d62-3baf-4860-8f46-d5f3849ac4cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:12.149199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.149169    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8211cfe6-7aee-43b9-b5f5-439ffb348931-kube-api-access-4zrpr" pod "8211cfe6-7aee-43b9-b5f5-439ffb348931" (UID: "8211cfe6-7aee-43b9-b5f5-439ffb348931"). InnerVolumeSpecName "kube-api-access-4zrpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:12.149529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.149506    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d984d62-3baf-4860-8f46-d5f3849ac4cd-kube-api-access-4pwtz" pod "5d984d62-3baf-4860-8f46-d5f3849ac4cd" (UID: "5d984d62-3baf-4860-8f46-d5f3849ac4cd"). InnerVolumeSpecName "kube-api-access-4pwtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:12.240141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.240107    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d984d62-3baf-4860-8f46-d5f3849ac4cd-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.240141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.240140    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8211cfe6-7aee-43b9-b5f5-439ffb348931-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.240141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.240150    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zrpr\" (UniqueName: \"kubernetes.io/projected/8211cfe6-7aee-43b9-b5f5-439ffb348931-kube-api-access-4zrpr\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.240457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.240160    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwtz\" (UniqueName: \"kubernetes.io/projected/5d984d62-3baf-4860-8f46-d5f3849ac4cd-kube-api-access-4pwtz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:12.583929 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:12.583889    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"]
Oct 31 01:49:13.028083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.026091    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"]
Oct 31 01:49:13.417073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.413440    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg" event={"ID":"8211cfe6-7aee-43b9-b5f5-439ffb348931","Type":"ContainerDied","Data":"e49ef85da29a7a44bcea2ed6e827eef5fb8746013ad847328f9b7e2964d96346"}
Oct 31 01:49:13.417073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.413531    5293 scope.go:122] "RemoveContainer" containerID="9eb6257b70fda2c3597230217ed844e85b6a44203ad7528d6c2317dd08231972"
Oct 31 01:49:13.417073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.413853    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"
Oct 31 01:49:13.455518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.450689    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9" event={"ID":"5d984d62-3baf-4860-8f46-d5f3849ac4cd","Type":"ContainerDied","Data":"4c969d03f60330c7e85016680a019399dd0e9fba188a44cc84aa460291a32ff7"}
Oct 31 01:49:13.455518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.451234    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"
Oct 31 01:49:13.471632 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.470116    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"
Oct 31 01:49:13.527926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.527900    5293 scope.go:122] "RemoveContainer" containerID="f73ecae6d9b76dca08a0919c26856bb4eefbef76a27f20d4c207efdbd37b70bd"
Oct 31 01:49:13.555743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.555712    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"]
Oct 31 01:49:13.558392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.558364    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d16e32-qwk89"]
Oct 31 01:49:13.578729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.578689    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"]
Oct 31 01:49:13.584639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.584616    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-b4565c8b9-h4chg"]
Oct 31 01:49:13.592947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.592920    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"]
Oct 31 01:49:13.597231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.597118    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-7d847948c9-sxfw9"]
Oct 31 01:49:13.600459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.600440    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"]
Oct 31 01:49:13.603803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:13.603756    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea6cc6-openshift-default-6bcf4bd495-vrwzt"]
Oct 31 01:49:14.176619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:14.176588    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5003abae-3098-4227-8b03-49f37e126502" path="/var/lib/kubelet/pods/5003abae-3098-4227-8b03-49f37e126502/volumes"
Oct 31 01:49:14.177097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:14.177083    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d984d62-3baf-4860-8f46-d5f3849ac4cd" path="/var/lib/kubelet/pods/5d984d62-3baf-4860-8f46-d5f3849ac4cd/volumes"
Oct 31 01:49:14.177458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:14.177444    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8211cfe6-7aee-43b9-b5f5-439ffb348931" path="/var/lib/kubelet/pods/8211cfe6-7aee-43b9-b5f5-439ffb348931/volumes"
Oct 31 01:49:14.178215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:14.178200    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" path="/var/lib/kubelet/pods/ad61337c-517a-4100-894d-d0169a6e69a1/volumes"
Oct 31 01:49:14.630395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:14.630314    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"]
Oct 31 01:49:14.630581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:14.630552    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s" podUID="840bd7db-af25-418d-964a-9640bb1bff43" containerName="istio-proxy" containerID="cri-o://b1580b64a145bec3ef503318d78bc1297acde8f58392c78b7494e728ea3834a7" gracePeriod=30
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.947553    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"]
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.948659    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5d984d62-3baf-4860-8f46-d5f3849ac4cd" containerName="payload-pre-processing"
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.948681    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d984d62-3baf-4860-8f46-d5f3849ac4cd" containerName="payload-pre-processing"
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.948711    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy"
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.948720    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy"
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.948791    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8211cfe6-7aee-43b9-b5f5-439ffb348931" containerName="payload-processing"
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.948801    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8211cfe6-7aee-43b9-b5f5-439ffb348931" containerName="payload-processing"
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.948867    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy"
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.948886    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy"
Oct 31 01:49:15.949114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.949113    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8211cfe6-7aee-43b9-b5f5-439ffb348931" containerName="payload-processing"
Oct 31 01:49:15.950109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.949134    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy"
Oct 31 01:49:15.950109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.949149    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d984d62-3baf-4860-8f46-d5f3849ac4cd" containerName="payload-pre-processing"
Oct 31 01:49:15.950109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.949159    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad61337c-517a-4100-894d-d0169a6e69a1" containerName="istio-proxy"
Oct 31 01:49:15.956094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.956011    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:15.971405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.965373    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-676801-openshift-default-dockercfg-tvvq7\""
Oct 31 01:49:15.981808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:15.981773    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"]
Oct 31 01:49:16.014936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.014896    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istiod-ca-cert\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.015158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.015099    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-podinfo\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.015239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.015218    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-token\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.015310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.015263    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-data\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.015379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.015351    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-envoy\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.015431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.015393    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-credential-socket\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.015492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.015475    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrnv\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-kube-api-access-mbrnv\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.015547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.015532    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-socket\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.015596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.015583    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-certs\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.118292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-data\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.118378    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-envoy\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.118416    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-credential-socket\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.118471    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrnv\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-kube-api-access-mbrnv\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.118531    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-socket\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.118587    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-certs\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.118672    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istiod-ca-cert\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.118795    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-podinfo\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.119475    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-token\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.120357    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-data\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.120925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.120859    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-certs\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.121362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.121112    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-socket\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.121734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.121712    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istiod-ca-cert\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.121984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.121965    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-credential-socket\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.122543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.122521    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-podinfo\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.124114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.123807    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-envoy\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.175075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.167416    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-token\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.175075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.171662    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrnv\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-kube-api-access-mbrnv\") pod \"e2e-api-a-676801-openshift-default-d68754d77-nljqb\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") " pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.293432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.293341    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:16.774380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.774353    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"]
Oct 31 01:49:16.774889 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:49:16.774869    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b7e1ab_0035_42d0_bb0f_41ba91d8a6f3.slice/crio-32a521e7b2f186063ab93f327ca9100a58abbe59b8087b1c91694ccdf6c2414b WatchSource:0}: Error finding container 32a521e7b2f186063ab93f327ca9100a58abbe59b8087b1c91694ccdf6c2414b: Status 404 returned error can't find the container with id 32a521e7b2f186063ab93f327ca9100a58abbe59b8087b1c91694ccdf6c2414b
Oct 31 01:49:16.777498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.777463    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:16.777621 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.777542    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:16.777621 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:16.777585    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:17.051937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.051847    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"]
Oct 31 01:49:17.058123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.058092    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6de735-5rl2c"]
Oct 31 01:49:17.172396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.172363    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"]
Oct 31 01:49:17.176871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.176850    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:17.193544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.193512    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cbbaec-dockercfg-ntngw\""
Oct 31 01:49:17.202525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.202273    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"]
Oct 31 01:49:17.208548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.208520    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"]
Oct 31 01:49:17.217349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.217326    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:17.242101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.239305    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"]
Oct 31 01:49:17.270531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.270496    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ac0383-2e25-4ef5-bae0-5372398f534e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b\" (UID: \"c5ac0383-2e25-4ef5-bae0-5372398f534e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:17.270826 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.270653    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a08ad4-f3cb-421f-963b-5e275efeb8c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn\" (UID: \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:17.270826 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.270711    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbplj\" (UniqueName: \"kubernetes.io/projected/46a08ad4-f3cb-421f-963b-5e275efeb8c6-kube-api-access-jbplj\") pod \"payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn\" (UID: \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:17.270960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.270870    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pzs\" (UniqueName: \"kubernetes.io/projected/c5ac0383-2e25-4ef5-bae0-5372398f534e-kube-api-access-g5pzs\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b\" (UID: \"c5ac0383-2e25-4ef5-bae0-5372398f534e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:17.292905 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.292875    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:49:17.376118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.373435    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ac0383-2e25-4ef5-bae0-5372398f534e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b\" (UID: \"c5ac0383-2e25-4ef5-bae0-5372398f534e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:17.376118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.373496    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a08ad4-f3cb-421f-963b-5e275efeb8c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn\" (UID: \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:17.376118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.373548    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbplj\" (UniqueName: \"kubernetes.io/projected/46a08ad4-f3cb-421f-963b-5e275efeb8c6-kube-api-access-jbplj\") pod \"payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn\" (UID: \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:17.376118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.373662    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pzs\" (UniqueName: \"kubernetes.io/projected/c5ac0383-2e25-4ef5-bae0-5372398f534e-kube-api-access-g5pzs\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b\" (UID: \"c5ac0383-2e25-4ef5-bae0-5372398f534e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:17.376447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.376246    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ac0383-2e25-4ef5-bae0-5372398f534e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b\" (UID: \"c5ac0383-2e25-4ef5-bae0-5372398f534e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:17.376990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.376959    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a08ad4-f3cb-421f-963b-5e275efeb8c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn\" (UID: \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:17.431104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.431073    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbplj\" (UniqueName: \"kubernetes.io/projected/46a08ad4-f3cb-421f-963b-5e275efeb8c6-kube-api-access-jbplj\") pod \"payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn\" (UID: \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:17.432250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.432202    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pzs\" (UniqueName: \"kubernetes.io/projected/c5ac0383-2e25-4ef5-bae0-5372398f534e-kube-api-access-g5pzs\") pod \"payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b\" (UID: \"c5ac0383-2e25-4ef5-bae0-5372398f534e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:17.500584 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.500306    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:17.551478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.551450    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:17.587156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.587114    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd" podUID="2d743ff0-4856-42bf-804b-ff2f686b2100" containerName="payload-processing" containerID="cri-o://d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949" gracePeriod=25
Oct 31 01:49:17.692705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.688838    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" event={"ID":"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3","Type":"ContainerStarted","Data":"06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3"}
Oct 31 01:49:17.692705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.689058    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" event={"ID":"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3","Type":"ContainerStarted","Data":"32a521e7b2f186063ab93f327ca9100a58abbe59b8087b1c91694ccdf6c2414b"}
Oct 31 01:49:17.741372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:17.740556    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podStartSLOduration=2.7405354219999998 podStartE2EDuration="2.740535422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:17.731425626 +0000 UTC m=+1715.680074018" watchObservedRunningTime="2026-10-31 01:49:17.740535422 +0000 UTC m=+1715.689183814"
Oct 31 01:49:18.027167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.027129    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr" podUID="677350f3-0c22-4e94-a120-870eec7a20b5" containerName="payload-pre-processing" containerID="cri-o://48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1" gracePeriod=25
Oct 31 01:49:18.068747 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.068714    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"]
Oct 31 01:49:18.113808 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:49:18.113772    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ac0383_2e25_4ef5_bae0_5372398f534e.slice/crio-5b07ea36d1309a224ee778c96cba200397579acffcc5a058f72386887386ede2 WatchSource:0}: Error finding container 5b07ea36d1309a224ee778c96cba200397579acffcc5a058f72386887386ede2: Status 404 returned error can't find the container with id 5b07ea36d1309a224ee778c96cba200397579acffcc5a058f72386887386ede2
Oct 31 01:49:18.182942 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.182877    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176cbbaa-756e-4daa-b257-7c38bd4463c7" path="/var/lib/kubelet/pods/176cbbaa-756e-4daa-b257-7c38bd4463c7/volumes"
Oct 31 01:49:18.270173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.269831    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:49:18.282014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.281970    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"]
Oct 31 01:49:18.283761 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:49:18.283724    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a08ad4_f3cb_421f_963b_5e275efeb8c6.slice/crio-54292973d25b7e5466c81a9c3eb2b8a24f946bd9b3603debb0c8bc845809a4b8 WatchSource:0}: Error finding container 54292973d25b7e5466c81a9c3eb2b8a24f946bd9b3603debb0c8bc845809a4b8: Status 404 returned error can't find the container with id 54292973d25b7e5466c81a9c3eb2b8a24f946bd9b3603debb0c8bc845809a4b8
Oct 31 01:49:18.302996 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.301152    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:18.330880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.320371    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:18.330880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.329717    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d743ff0-4856-42bf-804b-ff2f686b2100-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d743ff0-4856-42bf-804b-ff2f686b2100-plugins-config-volume\") pod \"2d743ff0-4856-42bf-804b-ff2f686b2100\" (UID: \"2d743ff0-4856-42bf-804b-ff2f686b2100\") "
Oct 31 01:49:18.330880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.329892    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d743ff0-4856-42bf-804b-ff2f686b2100-kube-api-access-7jrt7\" (UniqueName: \"kubernetes.io/projected/2d743ff0-4856-42bf-804b-ff2f686b2100-kube-api-access-7jrt7\") pod \"2d743ff0-4856-42bf-804b-ff2f686b2100\" (UID: \"2d743ff0-4856-42bf-804b-ff2f686b2100\") "
Oct 31 01:49:18.330880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.330556    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d743ff0-4856-42bf-804b-ff2f686b2100-plugins-config-volume" pod "2d743ff0-4856-42bf-804b-ff2f686b2100" (UID: "2d743ff0-4856-42bf-804b-ff2f686b2100"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:18.334463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.334436    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d743ff0-4856-42bf-804b-ff2f686b2100-kube-api-access-7jrt7" pod "2d743ff0-4856-42bf-804b-ff2f686b2100" (UID: "2d743ff0-4856-42bf-804b-ff2f686b2100"). InnerVolumeSpecName "kube-api-access-7jrt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:18.391334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.391239    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:49:18.431640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.431610    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677350f3-0c22-4e94-a120-870eec7a20b5-kube-api-access-8kzc5\" (UniqueName: \"kubernetes.io/projected/677350f3-0c22-4e94-a120-870eec7a20b5-kube-api-access-8kzc5\") pod \"677350f3-0c22-4e94-a120-870eec7a20b5\" (UID: \"677350f3-0c22-4e94-a120-870eec7a20b5\") "
Oct 31 01:49:18.432842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.431920    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677350f3-0c22-4e94-a120-870eec7a20b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677350f3-0c22-4e94-a120-870eec7a20b5-plugins-config-volume\") pod \"677350f3-0c22-4e94-a120-870eec7a20b5\" (UID: \"677350f3-0c22-4e94-a120-870eec7a20b5\") "
Oct 31 01:49:18.432842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.432372    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677350f3-0c22-4e94-a120-870eec7a20b5-plugins-config-volume" pod "677350f3-0c22-4e94-a120-870eec7a20b5" (UID: "677350f3-0c22-4e94-a120-870eec7a20b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:18.432842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.432745    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d743ff0-4856-42bf-804b-ff2f686b2100-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:18.432842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.432761    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jrt7\" (UniqueName: \"kubernetes.io/projected/2d743ff0-4856-42bf-804b-ff2f686b2100-kube-api-access-7jrt7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:18.432842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.432769    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/677350f3-0c22-4e94-a120-870eec7a20b5-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:18.436581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.436549    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677350f3-0c22-4e94-a120-870eec7a20b5-kube-api-access-8kzc5" pod "677350f3-0c22-4e94-a120-870eec7a20b5" (UID: "677350f3-0c22-4e94-a120-870eec7a20b5"). InnerVolumeSpecName "kube-api-access-8kzc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:18.536731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.536635    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kzc5\" (UniqueName: \"kubernetes.io/projected/677350f3-0c22-4e94-a120-870eec7a20b5-kube-api-access-8kzc5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:18.736880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.736837    5293 generic.go:356] "Generic (PLEG): container finished" podID="677350f3-0c22-4e94-a120-870eec7a20b5" containerID="48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1" exitCode=0
Oct 31 01:49:18.750311 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.750274    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"
Oct 31 01:49:18.753498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.753456    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr" event={"ID":"677350f3-0c22-4e94-a120-870eec7a20b5","Type":"ContainerDied","Data":"48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1"}
Oct 31 01:49:18.753623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.753542    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr" event={"ID":"677350f3-0c22-4e94-a120-870eec7a20b5","Type":"ContainerDied","Data":"915a4d3078c86d8ca0b9892f99f57b21926a597043fe1905d01c4440566142fd"}
Oct 31 01:49:18.753623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.753575    5293 scope.go:122] "RemoveContainer" containerID="48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1"
Oct 31 01:49:18.841275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.839652    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn" event={"ID":"46a08ad4-f3cb-421f-963b-5e275efeb8c6","Type":"ContainerStarted","Data":"0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726"}
Oct 31 01:49:18.841275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.839719    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn" event={"ID":"46a08ad4-f3cb-421f-963b-5e275efeb8c6","Type":"ContainerStarted","Data":"54292973d25b7e5466c81a9c3eb2b8a24f946bd9b3603debb0c8bc845809a4b8"}
Oct 31 01:49:18.841275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.840857    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:18.841275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.840967    5293 scope.go:122] "RemoveContainer" containerID="48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1"
Oct 31 01:49:18.847069 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:49:18.842392    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1\": container with ID starting with 48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1 not found: ID does not exist" containerID="48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1"
Oct 31 01:49:18.847069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.842429    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1"} err="failed to get container status \"48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1\": rpc error: code = NotFound desc = could not find container \"48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1\": container with ID starting with 48955e796dd0e8efb95425c27979a8cd8c590eb67251fa9c5fdaeebdf45d37b1 not found: ID does not exist"
Oct 31 01:49:18.862568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.862534    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b" event={"ID":"c5ac0383-2e25-4ef5-bae0-5372398f534e","Type":"ContainerStarted","Data":"ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db"}
Oct 31 01:49:18.862755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.862595    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b" event={"ID":"c5ac0383-2e25-4ef5-bae0-5372398f534e","Type":"ContainerStarted","Data":"5b07ea36d1309a224ee778c96cba200397579acffcc5a058f72386887386ede2"}
Oct 31 01:49:18.863252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.863225    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:18.865347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.865325    5293 generic.go:356] "Generic (PLEG): container finished" podID="2d743ff0-4856-42bf-804b-ff2f686b2100" containerID="d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949" exitCode=0
Oct 31 01:49:18.866467 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.866410    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"
Oct 31 01:49:18.867417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.867375    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn" podStartSLOduration=1.867360457 podStartE2EDuration="1.867360457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:18.866390585 +0000 UTC m=+1716.815038980" watchObservedRunningTime="2026-10-31 01:49:18.867360457 +0000 UTC m=+1716.816008850"
Oct 31 01:49:18.874625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.874601    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd" event={"ID":"2d743ff0-4856-42bf-804b-ff2f686b2100","Type":"ContainerDied","Data":"d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949"}
Oct 31 01:49:18.879004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.878301    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd" event={"ID":"2d743ff0-4856-42bf-804b-ff2f686b2100","Type":"ContainerDied","Data":"f64b6b040f646a07341b219794d68e1de72e0bd3c82ba87208210e9469c817bb"}
Oct 31 01:49:18.879004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.878346    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:18.879004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.878378    5293 scope.go:122] "RemoveContainer" containerID="d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949"
Oct 31 01:49:18.879666 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.879652    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:49:18.899396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.899280    5293 scope.go:122] "RemoveContainer" containerID="d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949"
Oct 31 01:49:18.900896 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:49:18.900832    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949\": container with ID starting with d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949 not found: ID does not exist" containerID="d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949"
Oct 31 01:49:18.900896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.900868    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949"} err="failed to get container status \"d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949\": rpc error: code = NotFound desc = could not find container \"d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949\": container with ID starting with d2310b839368042ab5d066957c202836f126c38943749e6a4d84abb641a07949 not found: ID does not exist"
Oct 31 01:49:18.902867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.902808    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"]
Oct 31 01:49:18.910494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.910470    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-5fbf9d6d6c-fnzlr"]
Oct 31 01:49:18.918890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.918845    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b" podStartSLOduration=1.918829532 podStartE2EDuration="1.918829532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:18.916371546 +0000 UTC m=+1716.865019941" watchObservedRunningTime="2026-10-31 01:49:18.918829532 +0000 UTC m=+1716.867477935"
Oct 31 01:49:18.933174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.933156    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"]
Oct 31 01:49:18.938596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:18.938579    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-7465d947c5-7h4qd"]
Oct 31 01:49:19.951618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:19.951542    5293 generic.go:356] "Generic (PLEG): container finished" podID="840bd7db-af25-418d-964a-9640bb1bff43" containerID="b1580b64a145bec3ef503318d78bc1297acde8f58392c78b7494e728ea3834a7" exitCode=0
Oct 31 01:49:19.952129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:19.951767    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s" event={"ID":"840bd7db-af25-418d-964a-9640bb1bff43","Type":"ContainerDied","Data":"b1580b64a145bec3ef503318d78bc1297acde8f58392c78b7494e728ea3834a7"}
Oct 31 01:49:20.568006 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.567966    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d743ff0-4856-42bf-804b-ff2f686b2100" path="/var/lib/kubelet/pods/2d743ff0-4856-42bf-804b-ff2f686b2100/volumes"
Oct 31 01:49:20.568766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.568743    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677350f3-0c22-4e94-a120-870eec7a20b5" path="/var/lib/kubelet/pods/677350f3-0c22-4e94-a120-870eec7a20b5/volumes"
Oct 31 01:49:20.573057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.573023    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:49:20.789559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.789469    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-certs\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.789983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.789963    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-credential-socket\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.790598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.790581    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-data\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.791029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.791009    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/840bd7db-af25-418d-964a-9640bb1bff43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840bd7db-af25-418d-964a-9640bb1bff43-istio-podinfo\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.791254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.791234    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-istio-token\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-istio-token\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.791788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.791769    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/840bd7db-af25-418d-964a-9640bb1bff43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840bd7db-af25-418d-964a-9640bb1bff43-istiod-ca-cert\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.791912 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.791898    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-socket\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.792118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.792102    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-envoy\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.792225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.792211    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-kube-api-access-ml4xk\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-kube-api-access-ml4xk\") pod \"840bd7db-af25-418d-964a-9640bb1bff43\" (UID: \"840bd7db-af25-418d-964a-9640bb1bff43\") "
Oct 31 01:49:20.794270 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.790480    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-credential-socket" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.794395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.790735    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-certs" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.794481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.791367    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-data" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.794560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.793843    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-socket" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.794628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.794246    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840bd7db-af25-418d-964a-9640bb1bff43-istiod-ca-cert" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:20.833277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.829180    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/840bd7db-af25-418d-964a-9640bb1bff43-istio-podinfo" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:49:20.833277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.829298    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-envoy" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.837685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.837648    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-kube-api-access-ml4xk" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "kube-api-access-ml4xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:20.843349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.843321    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-istio-token" pod "840bd7db-af25-418d-964a-9640bb1bff43" (UID: "840bd7db-af25-418d-964a-9640bb1bff43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:20.894342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894313    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.894342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894341    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.894342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894351    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.894578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894361    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840bd7db-af25-418d-964a-9640bb1bff43-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.894578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894374    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.894578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894386    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840bd7db-af25-418d-964a-9640bb1bff43-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.894578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894394    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.894578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894402    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840bd7db-af25-418d-964a-9640bb1bff43-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.894578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.894415    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4xk\" (UniqueName: \"kubernetes.io/projected/840bd7db-af25-418d-964a-9640bb1bff43-kube-api-access-ml4xk\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.963394 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.963364    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"
Oct 31 01:49:20.963830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.963391    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s" event={"ID":"840bd7db-af25-418d-964a-9640bb1bff43","Type":"ContainerDied","Data":"99317d1a79e47e63434f938689e4961d9d1b1e6fc5985e899aca71bf2ae315d9"}
Oct 31 01:49:20.963830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.963456    5293 scope.go:122] "RemoveContainer" containerID="b1580b64a145bec3ef503318d78bc1297acde8f58392c78b7494e728ea3834a7"
Oct 31 01:49:20.989665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:20.989640    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"]
Oct 31 01:49:21.001425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:21.001390    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-jwp5s"]
Oct 31 01:49:21.988879 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:21.968186    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:49:22.287858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:22.287766    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="840bd7db-af25-418d-964a-9640bb1bff43" path="/var/lib/kubelet/pods/840bd7db-af25-418d-964a-9640bb1bff43/volumes"
Oct 31 01:49:23.517302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:23.499261    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:49:26.046824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:26.046783    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"]
Oct 31 01:49:26.047458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:26.046854    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaff0ba-9mqb2"]
Oct 31 01:49:26.175812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:26.175767    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22bdab48-e65e-4dc0-94e9-e6d0458d8a90" path="/var/lib/kubelet/pods/22bdab48-e65e-4dc0-94e9-e6d0458d8a90/volumes"
Oct 31 01:49:27.025989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:27.025946    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"]
Oct 31 01:49:27.029435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:27.029406    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91311875-5w4nc"]
Oct 31 01:49:28.258659 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:28.258623    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8443aee4-f3ae-42f5-8e1d-2b7a10e052e7" path="/var/lib/kubelet/pods/8443aee4-f3ae-42f5-8e1d-2b7a10e052e7/volumes"
Oct 31 01:49:30.984090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:30.983197    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:49:30.984090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:30.983608    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:49:31.142513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.142488    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"]
Oct 31 01:49:31.143383 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.143366    5293 cpu_manager.go:414] "containerMap: removing container" podUID="677350f3-0c22-4e94-a120-870eec7a20b5" containerName="payload-pre-processing"
Oct 31 01:49:31.143511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.143500    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677350f3-0c22-4e94-a120-870eec7a20b5" containerName="payload-pre-processing"
Oct 31 01:49:31.143610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.143603    5293 cpu_manager.go:414] "containerMap: removing container" podUID="840bd7db-af25-418d-964a-9640bb1bff43" containerName="istio-proxy"
Oct 31 01:49:31.143661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.143654    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="840bd7db-af25-418d-964a-9640bb1bff43" containerName="istio-proxy"
Oct 31 01:49:31.143721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.143714    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2d743ff0-4856-42bf-804b-ff2f686b2100" containerName="payload-processing"
Oct 31 01:49:31.143768 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.143762    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d743ff0-4856-42bf-804b-ff2f686b2100" containerName="payload-processing"
Oct 31 01:49:31.143950 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.143942    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="840bd7db-af25-418d-964a-9640bb1bff43" containerName="istio-proxy"
Oct 31 01:49:31.144009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.144003    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d743ff0-4856-42bf-804b-ff2f686b2100" containerName="payload-processing"
Oct 31 01:49:31.144083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.144076    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="677350f3-0c22-4e94-a120-870eec7a20b5" containerName="payload-pre-processing"
Oct 31 01:49:31.150543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.150516    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.168570 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.168550    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-03004a\"/\"kube-root-ca.crt\""
Oct 31 01:49:31.168724 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.168591    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-03004a\"/\"openshift-service-ca.crt\""
Oct 31 01:49:31.172955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.172938    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-03004a\"/\"default-dockercfg-xmz9t\""
Oct 31 01:49:31.173202 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.173173    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-03004a\"/\"ipp-route-03004a-kserve-self-signed-certs\""
Oct 31 01:49:31.173292 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.173219    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"]
Oct 31 01:49:31.188515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.188470    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:49:31.189139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.188529    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:49:31.255271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.255196    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tls-certs\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.255421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.255353    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpss\" (UniqueName: \"kubernetes.io/projected/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-kube-api-access-ndpss\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.255421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.255402    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tmp-dir\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.255495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.255421    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-home\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.255495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.255439    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-model-cache\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.255647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.255627    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-dshm\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.357056 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.357006    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tls-certs\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.357424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.357392    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpss\" (UniqueName: \"kubernetes.io/projected/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-kube-api-access-ndpss\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.357574 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.357549    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tmp-dir\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.357695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.357671    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-home\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.357855 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.357708    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-model-cache\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.357993 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.357872    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-dshm\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.359423 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.359260    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tmp-dir\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.359914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.359847    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-home\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.362565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.362465    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-model-cache\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.363085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.363051    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tls-certs\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.365302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.365262    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-dshm\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.389891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.389417    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpss\" (UniqueName: \"kubernetes.io/projected/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-kube-api-access-ndpss\") pod \"ipp-route-03004a-kserve-8b8c57c85-v4pr7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") " pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:31.466251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:31.466216    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:32.015812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.015703    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"]
Oct 31 01:49:32.577875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.577827    5293 generic.go:356] "Generic (PLEG): container finished" podID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerID="cbb4796c756c3ad3ad41b6a2c3a4cb12604810c479c2deed9e5cf88f141c3edc" exitCode=0
Oct 31 01:49:32.578054 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.577877    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" event={"ID":"e1942154-283c-4d10-acb4-4a05d07a8ac5","Type":"ContainerDied","Data":"cbb4796c756c3ad3ad41b6a2c3a4cb12604810c479c2deed9e5cf88f141c3edc"}
Oct 31 01:49:32.578394 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.578376    5293 scope.go:122] "RemoveContainer" containerID="cbb4796c756c3ad3ad41b6a2c3a4cb12604810c479c2deed9e5cf88f141c3edc"
Oct 31 01:49:32.579878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.579848    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7" event={"ID":"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7","Type":"ContainerStarted","Data":"ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78"}
Oct 31 01:49:32.579962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.579888    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7" event={"ID":"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7","Type":"ContainerStarted","Data":"7ee2baec4c11bddfbb548729dc423ccbdb2927a93ed4a4b080a9ab3635f13497"}
Oct 31 01:49:32.580140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.580113    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:32.580192 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.580179    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:32.580232 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.580215    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:32.624496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:32.624141    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7" podStartSLOduration=1.624119624 podStartE2EDuration="1.624119624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:32.617528093 +0000 UTC m=+1730.566176498" watchObservedRunningTime="2026-10-31 01:49:32.624119624 +0000 UTC m=+1730.572768016"
Oct 31 01:49:33.467653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:33.467624    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:33.679723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:33.678969    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" event={"ID":"e1942154-283c-4d10-acb4-4a05d07a8ac5","Type":"ContainerStarted","Data":"eea4468a68755965fbf11171de155152fa220d42a33cf7eb8516ed325d503a78"}
Oct 31 01:49:34.079820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:34.079783    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:49:34.080003 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:34.079979    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 01:49:34.080093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:34.080022    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 01:49:35.082068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.079871    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 01:49:35.082068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.079942    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 01:49:35.153350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.153320    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"]
Oct 31 01:49:35.157745 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.157726    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.161012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.160983    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:49:35.162029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.161696    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:49:35.162029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.161709    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:49:35.162029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.161759    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:49:35.168588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.168569    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"]
Oct 31 01:49:35.234328 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.234282    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a152a14-0248-465d-b212-19082bceb9de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.234615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.234597    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a152a14-0248-465d-b212-19082bceb9de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.234836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.234808    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfz5\" (UniqueName: \"kubernetes.io/projected/7a152a14-0248-465d-b212-19082bceb9de-kube-api-access-qdfz5\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.337235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.337148    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfz5\" (UniqueName: \"kubernetes.io/projected/7a152a14-0248-465d-b212-19082bceb9de-kube-api-access-qdfz5\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.337531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.337511    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a152a14-0248-465d-b212-19082bceb9de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.337607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.337592    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a152a14-0248-465d-b212-19082bceb9de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.338470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.338443    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a152a14-0248-465d-b212-19082bceb9de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.340824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.340795    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a152a14-0248-465d-b212-19082bceb9de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.348707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.348680    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfz5\" (UniqueName: \"kubernetes.io/projected/7a152a14-0248-465d-b212-19082bceb9de-kube-api-access-qdfz5\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.475081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.475022    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:35.910366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:35.905327    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"]
Oct 31 01:49:36.079437 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:36.079402    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 01:49:36.079609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:36.079462    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 01:49:36.849117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:36.798108    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s" event={"ID":"7a152a14-0248-465d-b212-19082bceb9de","Type":"ContainerStarted","Data":"c142401263d60b18bca2f3e37fa8aefb6e4fc5cfee08fc5d10dcfb291ad429cb"}
Oct 31 01:49:36.849117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:36.848252    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s" event={"ID":"7a152a14-0248-465d-b212-19082bceb9de","Type":"ContainerStarted","Data":"67097e2565e9c4524e094de732b274ae21771e93a2fe9610190e87ebcf63a69b"}
Oct 31 01:49:36.866373 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:36.866316    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s" podStartSLOduration=1.866297375 podStartE2EDuration="1.866297375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:36.862982836 +0000 UTC m=+1734.811631229" watchObservedRunningTime="2026-10-31 01:49:36.866297375 +0000 UTC m=+1734.814945769"
Oct 31 01:49:37.035441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:37.035399    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"]
Oct 31 01:49:37.042256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:37.042228    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf054a0b-6qn92"]
Oct 31 01:49:37.080014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:37.079982    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 01:49:37.080177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:37.080057    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 01:49:37.836253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:37.836216    5293 generic.go:356] "Generic (PLEG): container finished" podID="7a152a14-0248-465d-b212-19082bceb9de" containerID="c142401263d60b18bca2f3e37fa8aefb6e4fc5cfee08fc5d10dcfb291ad429cb" exitCode=0
Oct 31 01:49:37.836460 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:37.836309    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s" event={"ID":"7a152a14-0248-465d-b212-19082bceb9de","Type":"ContainerDied","Data":"c142401263d60b18bca2f3e37fa8aefb6e4fc5cfee08fc5d10dcfb291ad429cb"}
Oct 31 01:49:38.079693 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:38.079659    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 01:49:38.080136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:38.079710    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 01:49:38.175885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:38.175845    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b96cbc5-8856-4892-9856-1775e4985c6b" path="/var/lib/kubelet/pods/9b96cbc5-8856-4892-9856-1775e4985c6b/volumes"
Oct 31 01:49:39.086067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.079908    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 01:49:39.086067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.079970    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 01:49:39.486936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.486648    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:39.508009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.507955    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a152a14-0248-465d-b212-19082bceb9de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a152a14-0248-465d-b212-19082bceb9de-maas-api-xks-ca\") pod \"7a152a14-0248-465d-b212-19082bceb9de\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") "
Oct 31 01:49:39.508477 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.508449    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a152a14-0248-465d-b212-19082bceb9de-kube-api-access-qdfz5\" (UniqueName: \"kubernetes.io/projected/7a152a14-0248-465d-b212-19082bceb9de-kube-api-access-qdfz5\") pod \"7a152a14-0248-465d-b212-19082bceb9de\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") "
Oct 31 01:49:39.508718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.508701    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a152a14-0248-465d-b212-19082bceb9de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a152a14-0248-465d-b212-19082bceb9de-maas-api-openshift-ca\") pod \"7a152a14-0248-465d-b212-19082bceb9de\" (UID: \"7a152a14-0248-465d-b212-19082bceb9de\") "
Oct 31 01:49:39.509899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.509876    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a152a14-0248-465d-b212-19082bceb9de-maas-api-openshift-ca" pod "7a152a14-0248-465d-b212-19082bceb9de" (UID: "7a152a14-0248-465d-b212-19082bceb9de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:39.512792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.512770    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a152a14-0248-465d-b212-19082bceb9de-maas-api-xks-ca" pod "7a152a14-0248-465d-b212-19082bceb9de" (UID: "7a152a14-0248-465d-b212-19082bceb9de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:49:39.512968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.512954    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a152a14-0248-465d-b212-19082bceb9de-kube-api-access-qdfz5" pod "7a152a14-0248-465d-b212-19082bceb9de" (UID: "7a152a14-0248-465d-b212-19082bceb9de"). InnerVolumeSpecName "kube-api-access-qdfz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:39.611513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.611413    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a152a14-0248-465d-b212-19082bceb9de-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:39.611513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.611473    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfz5\" (UniqueName: \"kubernetes.io/projected/7a152a14-0248-465d-b212-19082bceb9de-kube-api-access-qdfz5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:39.611513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.611488    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a152a14-0248-465d-b212-19082bceb9de-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:39.856523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.856495    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"
Oct 31 01:49:39.856711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.856536    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s" event={"ID":"7a152a14-0248-465d-b212-19082bceb9de","Type":"ContainerDied","Data":"67097e2565e9c4524e094de732b274ae21771e93a2fe9610190e87ebcf63a69b"}
Oct 31 01:49:39.856711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.856610    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67097e2565e9c4524e094de732b274ae21771e93a2fe9610190e87ebcf63a69b"
Oct 31 01:49:39.894759 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.894730    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"]
Oct 31 01:49:39.895503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.895480    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7a152a14-0248-465d-b212-19082bceb9de" containerName="revoke-keys"
Oct 31 01:49:39.895503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.895500    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a152a14-0248-465d-b212-19082bceb9de" containerName="revoke-keys"
Oct 31 01:49:39.895700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.895624    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a152a14-0248-465d-b212-19082bceb9de" containerName="revoke-keys"
Oct 31 01:49:39.899449 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.899429    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:39.916781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.916750    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tkj\" (UniqueName: \"kubernetes.io/projected/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-kube-api-access-h5tkj\") pod \"payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp\" (UID: \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:39.916951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.916899    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp\" (UID: \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:39.926299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.926273    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"]
Oct 31 01:49:39.966488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.966453    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-073e24-dockercfg-6zzn8\""
Oct 31 01:49:39.966936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:39.966911    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-073e24\""
Oct 31 01:49:40.077144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.066017    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp\" (UID: \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:40.077144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.066425    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tkj\" (UniqueName: \"kubernetes.io/projected/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-kube-api-access-h5tkj\") pod \"payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp\" (UID: \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:40.077144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.067648    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp\" (UID: \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:40.150913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.150882    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tkj\" (UniqueName: \"kubernetes.io/projected/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-kube-api-access-h5tkj\") pod \"payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp\" (UID: \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:40.234064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.230692    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:40.304071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.289217    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:49:40.304071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.289849    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:49:40.304071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.291777    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:49:40.657002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.656976    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"]
Oct 31 01:49:40.658219 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:49:40.658186    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f17b37_c55b_49c2_b5c1_b5c75228ce5a.slice/crio-3422854906ed58727dd22ac1edbc449e3ed24ccb0bd633d296f5e5305c6dc978 WatchSource:0}: Error finding container 3422854906ed58727dd22ac1edbc449e3ed24ccb0bd633d296f5e5305c6dc978: Status 404 returned error can't find the container with id 3422854906ed58727dd22ac1edbc449e3ed24ccb0bd633d296f5e5305c6dc978
Oct 31 01:49:40.866842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.866804    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp" event={"ID":"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a","Type":"ContainerStarted","Data":"758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82"}
Oct 31 01:49:40.866992 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.866863    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp" event={"ID":"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a","Type":"ContainerStarted","Data":"3422854906ed58727dd22ac1edbc449e3ed24ccb0bd633d296f5e5305c6dc978"}
Oct 31 01:49:40.867097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.867084    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:40.891509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:40.891460    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp" podStartSLOduration=1.8914459529999998 podStartE2EDuration="1.891445953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:40.889163013 +0000 UTC m=+1738.837811429" watchObservedRunningTime="2026-10-31 01:49:40.891445953 +0000 UTC m=+1738.840094345"
Oct 31 01:49:41.467661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:41.467571    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:41.485593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:41.485560    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:41.988696 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:49:41.988622    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:49:41.993428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:41.993405    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:49:43.334152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:43.334121    5293 scope.go:122] "RemoveContainer" containerID="63df7a9e5c522872b7aada955ea9b2072d8248057710978c22044b7453c89eba"
Oct 31 01:49:43.416186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:43.416163    5293 scope.go:122] "RemoveContainer" containerID="af3bf6abdb46239fd6047fb30e89cdcaefe5e43ff1aed96d7febcd20bbfbbfc9"
Oct 31 01:49:44.035321 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:44.035212    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"]
Oct 31 01:49:44.044767 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:44.044735    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"]
Oct 31 01:49:44.048349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:44.048324    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9125bc4-c9l7d"]
Oct 31 01:49:44.051489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:44.051466    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599d516a-r8gpn"]
Oct 31 01:49:44.177343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:44.177310    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99dd44dc-6599-4ec1-812d-d8e4b7144e09" path="/var/lib/kubelet/pods/99dd44dc-6599-4ec1-812d-d8e4b7144e09/volumes"
Oct 31 01:49:44.177925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:44.177909    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02964b6-a6cc-495d-98ee-9b8bff09cadc" path="/var/lib/kubelet/pods/d02964b6-a6cc-495d-98ee-9b8bff09cadc/volumes"
Oct 31 01:49:48.059150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:48.059112    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"]
Oct 31 01:49:48.066924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:48.066898    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1d037e2-kn5jx"]
Oct 31 01:49:48.176709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:48.176679    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99935a1b-b965-4ce1-999b-b414ec6ecae3" path="/var/lib/kubelet/pods/99935a1b-b965-4ce1-999b-b414ec6ecae3/volumes"
Oct 31 01:49:51.952626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:51.952597    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:49:52.255818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:52.255718    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"]
Oct 31 01:49:52.257930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:52.257622    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb" podUID="064d85d5-af98-4405-b040-dedf3c75e472" containerName="payload-pre-processing" containerID="cri-o://b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054" gracePeriod=30
Oct 31 01:49:52.815429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:52.813706    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"]
Oct 31 01:49:52.815429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:52.813768    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f0694-x7n6s"]
Oct 31 01:49:52.959656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:52.959630    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:49:53.172190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.172157    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064d85d5-af98-4405-b040-dedf3c75e472-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d85d5-af98-4405-b040-dedf3c75e472-plugins-config-volume\") pod \"064d85d5-af98-4405-b040-dedf3c75e472\" (UID: \"064d85d5-af98-4405-b040-dedf3c75e472\") "
Oct 31 01:49:53.172397 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.172270    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064d85d5-af98-4405-b040-dedf3c75e472-kube-api-access-m66m9\" (UniqueName: \"kubernetes.io/projected/064d85d5-af98-4405-b040-dedf3c75e472-kube-api-access-m66m9\") pod \"064d85d5-af98-4405-b040-dedf3c75e472\" (UID: \"064d85d5-af98-4405-b040-dedf3c75e472\") "
Oct 31 01:49:53.172573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.172547    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064d85d5-af98-4405-b040-dedf3c75e472-plugins-config-volume" pod "064d85d5-af98-4405-b040-dedf3c75e472" (UID: "064d85d5-af98-4405-b040-dedf3c75e472"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:53.173330 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.173104    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064d85d5-af98-4405-b040-dedf3c75e472-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:53.174704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.174679    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064d85d5-af98-4405-b040-dedf3c75e472-kube-api-access-m66m9" pod "064d85d5-af98-4405-b040-dedf3c75e472" (UID: "064d85d5-af98-4405-b040-dedf3c75e472"). InnerVolumeSpecName "kube-api-access-m66m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:53.274396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.274366    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m66m9\" (UniqueName: \"kubernetes.io/projected/064d85d5-af98-4405-b040-dedf3c75e472-kube-api-access-m66m9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:53.284198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.284167    5293 generic.go:356] "Generic (PLEG): container finished" podID="064d85d5-af98-4405-b040-dedf3c75e472" containerID="b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054" exitCode=0
Oct 31 01:49:53.284371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.284226    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"
Oct 31 01:49:53.284371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.284252    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb" event={"ID":"064d85d5-af98-4405-b040-dedf3c75e472","Type":"ContainerDied","Data":"b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054"}
Oct 31 01:49:53.284371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.284312    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb" event={"ID":"064d85d5-af98-4405-b040-dedf3c75e472","Type":"ContainerDied","Data":"4dab8851b7aa307e8cf2bad46fb018310d96e7cc8516b5b1391a00d4cbecc30b"}
Oct 31 01:49:53.284371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.284339    5293 scope.go:122] "RemoveContainer" containerID="b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054"
Oct 31 01:49:53.295806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.295783    5293 scope.go:122] "RemoveContainer" containerID="b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054"
Oct 31 01:49:53.296188 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:49:53.296158    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054\": container with ID starting with b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054 not found: ID does not exist" containerID="b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054"
Oct 31 01:49:53.296490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.296197    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054"} err="failed to get container status \"b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054\": rpc error: code = NotFound desc = could not find container \"b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054\": container with ID starting with b6097f5c2d7b55c2e2fa53549fd9fb6c5d5cb9d4927a18d9f3f6ac6a4c935054 not found: ID does not exist"
Oct 31 01:49:53.313981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.313921    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"]
Oct 31 01:49:53.319637 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:53.319591    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-867ff7997f-plspb"]
Oct 31 01:49:54.178580 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:54.178543    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064d85d5-af98-4405-b040-dedf3c75e472" path="/var/lib/kubelet/pods/064d85d5-af98-4405-b040-dedf3c75e472/volumes"
Oct 31 01:49:54.179294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:54.179257    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a152a14-0248-465d-b212-19082bceb9de" path="/var/lib/kubelet/pods/7a152a14-0248-465d-b212-19082bceb9de/volumes"
Oct 31 01:49:55.910269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:55.910192    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:49:55.910932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:55.910893    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:49:57.069549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:57.069508    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"]
Oct 31 01:49:57.087293 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:57.086479    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2e958-9mqnv"]
Oct 31 01:49:57.338004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:57.336246    5293 generic.go:356] "Generic (PLEG): container finished" podID="5a96b284-a772-4d38-aac9-665181786781" containerID="80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d" exitCode=0
Oct 31 01:49:57.338004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:57.336424    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" event={"ID":"5a96b284-a772-4d38-aac9-665181786781","Type":"ContainerDied","Data":"80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d"}
Oct 31 01:49:57.338004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:57.337000    5293 scope.go:122] "RemoveContainer" containerID="80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d"
Oct 31 01:49:57.341199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:57.340345    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:57.341199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:57.340451    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:57.341199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:57.340502    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:58.175260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.175220    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b23f00-d945-478d-953d-47498e96df27" path="/var/lib/kubelet/pods/c5b23f00-d945-478d-953d-47498e96df27/volumes"
Oct 31 01:49:58.347784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.347748    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" event={"ID":"5a96b284-a772-4d38-aac9-665181786781","Type":"ContainerStarted","Data":"cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514"}
Oct 31 01:49:58.349767 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.349741    5293 generic.go:356] "Generic (PLEG): container finished" podID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerID="06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3" exitCode=0
Oct 31 01:49:58.349908 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.349805    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" event={"ID":"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3","Type":"ContainerDied","Data":"06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3"}
Oct 31 01:49:58.350261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.350245    5293 scope.go:122] "RemoveContainer" containerID="06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3"
Oct 31 01:49:58.351965 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.351932    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:58.352100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.352017    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:58.352100 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.352092    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:49:58.857117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.856406    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:49:58.857117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.856555    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:49:58.857117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:58.856610    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 01:49:59.472087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:59.471668    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" event={"ID":"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3","Type":"ContainerStarted","Data":"c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3"}
Oct 31 01:49:59.856124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:59.855442    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:49:59.856124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:49:59.855521    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 01:50:00.307065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:00.297751    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:50:00.307065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:00.297930    5293 patch_prober.go:29] interesting pod/e2e-api-a-676801-openshift-default-d68754d77-nljqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:50:00.307065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:00.297987    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:50:00.861062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:00.858263    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:50:00.861062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:00.858338    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 01:50:01.158357 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.158321    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"]
Oct 31 01:50:01.158937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.158875    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy" containerID="cri-o://78d2efcf868ff1117db4c3cd0ee38f56b3e28ce7d57a136f5bc003f2254bb0de" gracePeriod=30
Oct 31 01:50:01.314065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.300434    5293 patch_prober.go:29] interesting pod/e2e-api-a-676801-openshift-default-d68754d77-nljqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:50:01.336078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.314346    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:50:01.764063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.759008    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"]
Oct 31 01:50:01.764063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.760108    5293 cpu_manager.go:414] "containerMap: removing container" podUID="064d85d5-af98-4405-b040-dedf3c75e472" containerName="payload-pre-processing"
Oct 31 01:50:01.764063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.760131    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064d85d5-af98-4405-b040-dedf3c75e472" containerName="payload-pre-processing"
Oct 31 01:50:01.764063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.760394    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="064d85d5-af98-4405-b040-dedf3c75e472" containerName="payload-pre-processing"
Oct 31 01:50:01.792064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.791218    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"]
Oct 31 01:50:01.792064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.791339    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:01.880195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.837719    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:50:01.880195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.837881    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:50:01.880195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.837969    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:50:01.880195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.838660    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:50:01.880195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.868225    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4d98\" (UniqueName: \"kubernetes.io/projected/9bedd736-5a32-460b-967a-ef94e208a02d-kube-api-access-m4d98\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:01.880195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.868311    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:01.880195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.868503    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:01.926941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.887835    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:50:01.926941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.887912    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 01:50:02.003066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.975205    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4d98\" (UniqueName: \"kubernetes.io/projected/9bedd736-5a32-460b-967a-ef94e208a02d-kube-api-access-m4d98\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:02.003066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.975315    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:02.003066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.975408    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:02.003066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:01.976857    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:02.098130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:02.079666    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4d98\" (UniqueName: \"kubernetes.io/projected/9bedd736-5a32-460b-967a-ef94e208a02d-kube-api-access-m4d98\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:02.098367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:02.084026    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:02.295490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:02.295455    5293 patch_prober.go:29] interesting pod/e2e-api-a-676801-openshift-default-d68754d77-nljqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:50:02.295651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:02.295522    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:50:02.360898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:02.355788    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:02.876555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:02.871154    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:50:02.876555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:02.871224    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 01:50:02.879272 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:02.879244    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"]
Oct 31 01:50:02.879529 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:50:02.879489    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bedd736_5a32_460b_967a_ef94e208a02d.slice/crio-500865bef6e59ef2693198896e359ea18334ef4a500ec688edde06698fcc06fd WatchSource:0}: Error finding container 500865bef6e59ef2693198896e359ea18334ef4a500ec688edde06698fcc06fd: Status 404 returned error can't find the container with id 500865bef6e59ef2693198896e359ea18334ef4a500ec688edde06698fcc06fd
Oct 31 01:50:03.294912 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.294877    5293 patch_prober.go:29] interesting pod/e2e-api-a-676801-openshift-default-d68754d77-nljqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:50:03.295319 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.294931    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:50:03.497508 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.497476    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"]
Oct 31 01:50:03.578439 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.578351    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"]
Oct 31 01:50:03.578799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.578778    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:03.580220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.580193    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"]
Oct 31 01:50:03.583745 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.583723    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-90c4d1-dockercfg-sssq4\""
Oct 31 01:50:03.584661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.583948    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-90c4d1\""
Oct 31 01:50:03.591206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.591135    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"]
Oct 31 01:50:03.591308 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.591240    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:03.767115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.722057    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh\" (UID: \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:03.767115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.722250    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg4vn\" (UniqueName: \"kubernetes.io/projected/04baaad8-eb5e-4421-9fd8-9e77f229b5af-kube-api-access-fg4vn\") pod \"payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h\" (UID: \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:03.767115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.722350    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04baaad8-eb5e-4421-9fd8-9e77f229b5af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h\" (UID: \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:03.767115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.722419    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwbd\" (UniqueName: \"kubernetes.io/projected/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-kube-api-access-frwbd\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh\" (UID: \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:03.868484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.837929    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh\" (UID: \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:03.868484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.838099    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg4vn\" (UniqueName: \"kubernetes.io/projected/04baaad8-eb5e-4421-9fd8-9e77f229b5af-kube-api-access-fg4vn\") pod \"payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h\" (UID: \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:03.868484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.838179    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04baaad8-eb5e-4421-9fd8-9e77f229b5af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h\" (UID: \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:03.868484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.838234    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwbd\" (UniqueName: \"kubernetes.io/projected/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-kube-api-access-frwbd\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh\" (UID: \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:03.890986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.872603    5293 patch_prober.go:29] interesting pod/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:50:03.890986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.872656    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 01:50:03.898334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.891931    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:50:03.898476 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.898432    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg4vn\" (UniqueName: \"kubernetes.io/projected/04baaad8-eb5e-4421-9fd8-9e77f229b5af-kube-api-access-fg4vn\") pod \"payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h\" (UID: \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:03.934153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.914914    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh\" (UID: \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:03.934153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:03.928302    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04baaad8-eb5e-4421-9fd8-9e77f229b5af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h\" (UID: \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:04.129388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.109068    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:04.285338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.285306    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwbd\" (UniqueName: \"kubernetes.io/projected/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-kube-api-access-frwbd\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh\" (UID: \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:04.286701 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.286668    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv" event={"ID":"9bedd736-5a32-460b-967a-ef94e208a02d","Type":"ContainerStarted","Data":"1298ef85051a60bffa36edb61e157bba1477a974622ee0cd4762375c7911a024"}
Oct 31 01:50:04.286816 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.286730    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv" event={"ID":"9bedd736-5a32-460b-967a-ef94e208a02d","Type":"ContainerStarted","Data":"500865bef6e59ef2693198896e359ea18334ef4a500ec688edde06698fcc06fd"}
Oct 31 01:50:04.295970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.295942    5293 patch_prober.go:29] interesting pod/e2e-api-a-676801-openshift-default-d68754d77-nljqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:50:04.296517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.296488    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:50:04.389976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.386299    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:04.577441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.576160    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv" podStartSLOduration=3.57613928 podStartE2EDuration="3.57613928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:04.310359669 +0000 UTC m=+1762.259008077" watchObservedRunningTime="2026-10-31 01:50:04.57613928 +0000 UTC m=+1762.524787674"
Oct 31 01:50:04.577441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:04.576737    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"]
Oct 31 01:50:05.042062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.033975    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"]
Oct 31 01:50:05.094962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.094923    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:50:05.095531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.095465    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:50:05.127062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.123522    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:50:05.295700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.294364    5293 patch_prober.go:29] interesting pod/e2e-api-a-676801-openshift-default-d68754d77-nljqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:50:05.295700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.294424    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:50:05.407114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.406840    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h" event={"ID":"04baaad8-eb5e-4421-9fd8-9e77f229b5af","Type":"ContainerStarted","Data":"4357ea3d00c8e8cd812e65c83e3f87aae8786aeab60aaccbd5d64a521ec6a011"}
Oct 31 01:50:05.407114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.406904    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h" event={"ID":"04baaad8-eb5e-4421-9fd8-9e77f229b5af","Type":"ContainerStarted","Data":"35ea9cd832bcff31d9334392e8fc5eb57c2a344c0f104e286b70cf74dddc921a"}
Oct 31 01:50:05.428514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.428485    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:05.476685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.455551    5293 generic.go:356] "Generic (PLEG): container finished" podID="9bedd736-5a32-460b-967a-ef94e208a02d" containerID="1298ef85051a60bffa36edb61e157bba1477a974622ee0cd4762375c7911a024" exitCode=0
Oct 31 01:50:05.476685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.455824    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv" event={"ID":"9bedd736-5a32-460b-967a-ef94e208a02d","Type":"ContainerDied","Data":"1298ef85051a60bffa36edb61e157bba1477a974622ee0cd4762375c7911a024"}
Oct 31 01:50:05.545544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.531548    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh" event={"ID":"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2","Type":"ContainerStarted","Data":"54fd4b480c44ee14376cb537e029415add21e9451a57f972dc0bf569ca4b966b"}
Oct 31 01:50:05.545544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.531618    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh" event={"ID":"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2","Type":"ContainerStarted","Data":"91e22a1c93dbe75434c36e2a6e7e2c0c27743736dbf6c1098e36d7eb9a1458d7"}
Oct 31 01:50:05.545544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.537429    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:05.568651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.568582    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h" podStartSLOduration=2.568560746 podStartE2EDuration="2.568560746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:05.508468171 +0000 UTC m=+1763.457116565" watchObservedRunningTime="2026-10-31 01:50:05.568560746 +0000 UTC m=+1763.517209135"
Oct 31 01:50:05.758442 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:05.758323    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh" podStartSLOduration=2.758300228 podStartE2EDuration="2.758300228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:05.666549732 +0000 UTC m=+1763.615198124" watchObservedRunningTime="2026-10-31 01:50:05.758300228 +0000 UTC m=+1763.706948621"
Oct 31 01:50:06.089696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.089596    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"]
Oct 31 01:50:06.095581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.095549    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e20db3-m4qkx"]
Oct 31 01:50:06.181389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.181354    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863a76e1-6afc-4434-8b77-4f46f4b86be5" path="/var/lib/kubelet/pods/863a76e1-6afc-4434-8b77-4f46f4b86be5/volumes"
Oct 31 01:50:06.299561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.294623    5293 patch_prober.go:29] interesting pod/e2e-api-a-676801-openshift-default-d68754d77-nljqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:50:06.299561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.294675    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:50:06.508466 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.508424    5293 generic.go:356] "Generic (PLEG): container finished" podID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerID="78d2efcf868ff1117db4c3cd0ee38f56b3e28ce7d57a136f5bc003f2254bb0de" exitCode=0
Oct 31 01:50:06.508981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.508874    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" event={"ID":"546bcf87-5caf-4136-8a1f-d6719ba577cb","Type":"ContainerDied","Data":"78d2efcf868ff1117db4c3cd0ee38f56b3e28ce7d57a136f5bc003f2254bb0de"}
Oct 31 01:50:06.508981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.508947    5293 scope.go:122] "RemoveContainer" containerID="2e1cbff65f4405c2a8cecd8ac5bb497a488abb505635869b31a7335ac22bf817"
Oct 31 01:50:06.632644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.632607    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:50:06.718944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.718801    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/546bcf87-5caf-4136-8a1f-d6719ba577cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/546bcf87-5caf-4136-8a1f-d6719ba577cb-istiod-ca-cert\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.719123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.718919    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-podinfo\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.719123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.719008    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-envoy\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.719429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.719161    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-credential-socket\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.719429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.719206    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-certs\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.719429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.719293    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-kube-api-access-wsc4p\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-kube-api-access-wsc4p\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.719429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.719390    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-socket\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.719649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.719541    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-token\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-token\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.719649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.719593    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-data\") pod \"546bcf87-5caf-4136-8a1f-d6719ba577cb\" (UID: \"546bcf87-5caf-4136-8a1f-d6719ba577cb\") "
Oct 31 01:50:06.721271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.721241    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-certs" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:06.721574 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.721552    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-socket" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:06.722285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.722260    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-credential-socket" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:06.722636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.722612    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-data" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:06.723866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.723835    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546bcf87-5caf-4136-8a1f-d6719ba577cb-istiod-ca-cert" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:06.724032 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.724007    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.724635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.724613    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.724731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.724640    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.724731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.724655    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.724731 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.724577    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-podinfo" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:06.727968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.727939    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-kube-api-access-wsc4p" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "kube-api-access-wsc4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:06.730649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.730621    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-envoy" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:06.734609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.734569    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-token" pod "546bcf87-5caf-4136-8a1f-d6719ba577cb" (UID: "546bcf87-5caf-4136-8a1f-d6719ba577cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:06.762170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.762135    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:06.826129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.826094    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bedd736-5a32-460b-967a-ef94e208a02d-kube-api-access-m4d98\" (UniqueName: \"kubernetes.io/projected/9bedd736-5a32-460b-967a-ef94e208a02d-kube-api-access-m4d98\") pod \"9bedd736-5a32-460b-967a-ef94e208a02d\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") "
Oct 31 01:50:06.826335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.826183    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-openshift-ca\") pod \"9bedd736-5a32-460b-967a-ef94e208a02d\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") "
Oct 31 01:50:06.826335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.826310    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-xks-ca\") pod \"9bedd736-5a32-460b-967a-ef94e208a02d\" (UID: \"9bedd736-5a32-460b-967a-ef94e208a02d\") "
Oct 31 01:50:06.826656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.826619    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-openshift-ca" pod "9bedd736-5a32-460b-967a-ef94e208a02d" (UID: "9bedd736-5a32-460b-967a-ef94e208a02d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:06.827943 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.827913    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.828079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.827947    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.828079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.827961    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/546bcf87-5caf-4136-8a1f-d6719ba577cb-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.828079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.827975    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.828079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.827987    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/546bcf87-5caf-4136-8a1f-d6719ba577cb-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.828079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.828000    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsc4p\" (UniqueName: \"kubernetes.io/projected/546bcf87-5caf-4136-8a1f-d6719ba577cb-kube-api-access-wsc4p\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.828833 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.828810    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bedd736-5a32-460b-967a-ef94e208a02d-kube-api-access-m4d98" pod "9bedd736-5a32-460b-967a-ef94e208a02d" (UID: "9bedd736-5a32-460b-967a-ef94e208a02d"). InnerVolumeSpecName "kube-api-access-m4d98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:06.829015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.829000    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-xks-ca" pod "9bedd736-5a32-460b-967a-ef94e208a02d" (UID: "9bedd736-5a32-460b-967a-ef94e208a02d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:06.930098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.930061    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4d98\" (UniqueName: \"kubernetes.io/projected/9bedd736-5a32-460b-967a-ef94e208a02d-kube-api-access-m4d98\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:06.930098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:06.930097    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bedd736-5a32-460b-967a-ef94e208a02d-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:07.299348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.299317    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:50:07.300669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.300647    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:50:07.521514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.521427    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"
Oct 31 01:50:07.521514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.521451    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv" event={"ID":"9bedd736-5a32-460b-967a-ef94e208a02d","Type":"ContainerDied","Data":"500865bef6e59ef2693198896e359ea18334ef4a500ec688edde06698fcc06fd"}
Oct 31 01:50:07.521514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.521512    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="500865bef6e59ef2693198896e359ea18334ef4a500ec688edde06698fcc06fd"
Oct 31 01:50:07.523375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.523357    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"
Oct 31 01:50:07.523523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.523402    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6" event={"ID":"546bcf87-5caf-4136-8a1f-d6719ba577cb","Type":"ContainerDied","Data":"495a02e24249196511ee6c34275df095247a940744cc52a421d5db630621d33b"}
Oct 31 01:50:07.523523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.523452    5293 scope.go:122] "RemoveContainer" containerID="78d2efcf868ff1117db4c3cd0ee38f56b3e28ce7d57a136f5bc003f2254bb0de"
Oct 31 01:50:07.556935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.556912    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"]
Oct 31 01:50:07.560683 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:07.560664    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f0694-openshift-default-5d464cbc67-47sm6"]
Oct 31 01:50:08.175653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:08.175622    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" path="/var/lib/kubelet/pods/546bcf87-5caf-4136-8a1f-d6719ba577cb/volumes"
Oct 31 01:50:11.379260 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.379213    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"]
Oct 31 01:50:11.380342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.380317    5293 cpu_manager.go:414] "containerMap: removing container" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy"
Oct 31 01:50:11.380342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.380343    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy"
Oct 31 01:50:11.380529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.380392    5293 cpu_manager.go:414] "containerMap: removing container" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy"
Oct 31 01:50:11.380529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.380401    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy"
Oct 31 01:50:11.380529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.380432    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9bedd736-5a32-460b-967a-ef94e208a02d" containerName="revoke-keys"
Oct 31 01:50:11.380529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.380440    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bedd736-5a32-460b-967a-ef94e208a02d" containerName="revoke-keys"
Oct 31 01:50:11.380777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.380628    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy"
Oct 31 01:50:11.380777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.380650    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bedd736-5a32-460b-967a-ef94e208a02d" containerName="revoke-keys"
Oct 31 01:50:11.392625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.387891    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.392625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.391353    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"]
Oct 31 01:50:11.392625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.391989    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:50:11.392625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.392184    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:50:11.403410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.402570    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:50:11.403410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.402661    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:50:11.568077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.560532    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.568077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.560681    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6sg\" (UniqueName: \"kubernetes.io/projected/80424e2c-21e7-430d-9f3b-9449b045fef2-kube-api-access-jv6sg\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.568077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.560876    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.667114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.664275    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6sg\" (UniqueName: \"kubernetes.io/projected/80424e2c-21e7-430d-9f3b-9449b045fef2-kube-api-access-jv6sg\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.667114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.664418    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.667114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.664694    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.667114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.665823    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.669531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.669501    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.683063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.679124    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6sg\" (UniqueName: \"kubernetes.io/projected/80424e2c-21e7-430d-9f3b-9449b045fef2-kube-api-access-jv6sg\") pod \"maas-api-revoke-keys-e2e-api-a-676801-flbcj\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.720304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:11.720269    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:11.787581 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:50:11.787546    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:50:12.053470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:12.053384    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"]
Oct 31 01:50:12.096903 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:50:12.096867    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80424e2c_21e7_430d_9f3b_9449b045fef2.slice/crio-ec91e271e37921ad16f0be43ce5a74a8e5c507dc2a9d209d328b475a929aa997 WatchSource:0}: Error finding container ec91e271e37921ad16f0be43ce5a74a8e5c507dc2a9d209d328b475a929aa997: Status 404 returned error can't find the container with id ec91e271e37921ad16f0be43ce5a74a8e5c507dc2a9d209d328b475a929aa997
Oct 31 01:50:12.654217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:12.654175    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj" event={"ID":"80424e2c-21e7-430d-9f3b-9449b045fef2","Type":"ContainerStarted","Data":"ec91e271e37921ad16f0be43ce5a74a8e5c507dc2a9d209d328b475a929aa997"}
Oct 31 01:50:13.755495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:13.755460    5293 generic.go:356] "Generic (PLEG): container finished" podID="80424e2c-21e7-430d-9f3b-9449b045fef2" containerID="b8622d5e86f700778121b9e88f2b52f6204d93b231060ad7d37a088ed3958370" exitCode=0
Oct 31 01:50:13.756066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:13.755837    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj" event={"ID":"80424e2c-21e7-430d-9f3b-9449b045fef2","Type":"ContainerDied","Data":"b8622d5e86f700778121b9e88f2b52f6204d93b231060ad7d37a088ed3958370"}
Oct 31 01:50:14.433478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:14.433430    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:50:15.076256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.076234    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:15.166652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.166617    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-xks-ca\") pod \"80424e2c-21e7-430d-9f3b-9449b045fef2\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") "
Oct 31 01:50:15.166817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.166779    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80424e2c-21e7-430d-9f3b-9449b045fef2-kube-api-access-jv6sg\" (UniqueName: \"kubernetes.io/projected/80424e2c-21e7-430d-9f3b-9449b045fef2-kube-api-access-jv6sg\") pod \"80424e2c-21e7-430d-9f3b-9449b045fef2\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") "
Oct 31 01:50:15.166961 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.166944    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-openshift-ca\") pod \"80424e2c-21e7-430d-9f3b-9449b045fef2\" (UID: \"80424e2c-21e7-430d-9f3b-9449b045fef2\") "
Oct 31 01:50:15.167509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.167322    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-openshift-ca" pod "80424e2c-21e7-430d-9f3b-9449b045fef2" (UID: "80424e2c-21e7-430d-9f3b-9449b045fef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:15.167747 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.167653    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:15.169088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.169066    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-xks-ca" pod "80424e2c-21e7-430d-9f3b-9449b045fef2" (UID: "80424e2c-21e7-430d-9f3b-9449b045fef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:15.169220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.169203    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80424e2c-21e7-430d-9f3b-9449b045fef2-kube-api-access-jv6sg" pod "80424e2c-21e7-430d-9f3b-9449b045fef2" (UID: "80424e2c-21e7-430d-9f3b-9449b045fef2"). InnerVolumeSpecName "kube-api-access-jv6sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:15.269527 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.269496    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80424e2c-21e7-430d-9f3b-9449b045fef2-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:15.269527 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.269525    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv6sg\" (UniqueName: \"kubernetes.io/projected/80424e2c-21e7-430d-9f3b-9449b045fef2-kube-api-access-jv6sg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:15.819984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.819928    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj" event={"ID":"80424e2c-21e7-430d-9f3b-9449b045fef2","Type":"ContainerDied","Data":"ec91e271e37921ad16f0be43ce5a74a8e5c507dc2a9d209d328b475a929aa997"}
Oct 31 01:50:15.820213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.820010    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec91e271e37921ad16f0be43ce5a74a8e5c507dc2a9d209d328b475a929aa997"
Oct 31 01:50:15.820213 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:15.820141    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"
Oct 31 01:50:16.562816 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:16.549507    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:16.618063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:16.605240    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:17.813910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:17.813864    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"]
Oct 31 01:50:17.814430 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:17.814180    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn" podUID="46a08ad4-f3cb-421f-963b-5e275efeb8c6" containerName="payload-processing" containerID="cri-o://0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726" gracePeriod=30
Oct 31 01:50:17.822461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:17.822434    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"]
Oct 31 01:50:17.822725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:17.822685    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b" podUID="c5ac0383-2e25-4ef5-bae0-5372398f534e" containerName="payload-pre-processing" containerID="cri-o://ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db" gracePeriod=30
Oct 31 01:50:18.353067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.347444    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"]
Oct 31 01:50:18.619818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.613213    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbbaec-45hfv"]
Oct 31 01:50:18.643110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.640454    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:50:18.697030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.697003    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:50:18.767793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.767755    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a08ad4-f3cb-421f-963b-5e275efeb8c6-kube-api-access-jbplj\" (UniqueName: \"kubernetes.io/projected/46a08ad4-f3cb-421f-963b-5e275efeb8c6-kube-api-access-jbplj\") pod \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\" (UID: \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\") "
Oct 31 01:50:18.767992 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.767908    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a08ad4-f3cb-421f-963b-5e275efeb8c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a08ad4-f3cb-421f-963b-5e275efeb8c6-plugins-config-volume\") pod \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\" (UID: \"46a08ad4-f3cb-421f-963b-5e275efeb8c6\") "
Oct 31 01:50:18.768087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.768072    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ac0383-2e25-4ef5-bae0-5372398f534e-kube-api-access-g5pzs\" (UniqueName: \"kubernetes.io/projected/c5ac0383-2e25-4ef5-bae0-5372398f534e-kube-api-access-g5pzs\") pod \"c5ac0383-2e25-4ef5-bae0-5372398f534e\" (UID: \"c5ac0383-2e25-4ef5-bae0-5372398f534e\") "
Oct 31 01:50:18.768144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.768130    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ac0383-2e25-4ef5-bae0-5372398f534e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ac0383-2e25-4ef5-bae0-5372398f534e-plugins-config-volume\") pod \"c5ac0383-2e25-4ef5-bae0-5372398f534e\" (UID: \"c5ac0383-2e25-4ef5-bae0-5372398f534e\") "
Oct 31 01:50:18.768831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.768804    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ac0383-2e25-4ef5-bae0-5372398f534e-plugins-config-volume" pod "c5ac0383-2e25-4ef5-bae0-5372398f534e" (UID: "c5ac0383-2e25-4ef5-bae0-5372398f534e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:18.768831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.768584    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a08ad4-f3cb-421f-963b-5e275efeb8c6-plugins-config-volume" pod "46a08ad4-f3cb-421f-963b-5e275efeb8c6" (UID: "46a08ad4-f3cb-421f-963b-5e275efeb8c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:18.769135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.769117    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ac0383-2e25-4ef5-bae0-5372398f534e-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:18.769135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.769135    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a08ad4-f3cb-421f-963b-5e275efeb8c6-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:18.770396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.770367    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ac0383-2e25-4ef5-bae0-5372398f534e-kube-api-access-g5pzs" pod "c5ac0383-2e25-4ef5-bae0-5372398f534e" (UID: "c5ac0383-2e25-4ef5-bae0-5372398f534e"). InnerVolumeSpecName "kube-api-access-g5pzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:18.770518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.770499    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a08ad4-f3cb-421f-963b-5e275efeb8c6-kube-api-access-jbplj" pod "46a08ad4-f3cb-421f-963b-5e275efeb8c6" (UID: "46a08ad4-f3cb-421f-963b-5e275efeb8c6"). InnerVolumeSpecName "kube-api-access-jbplj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:18.863882 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.863852    5293 generic.go:356] "Generic (PLEG): container finished" podID="46a08ad4-f3cb-421f-963b-5e275efeb8c6" containerID="0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726" exitCode=0
Oct 31 01:50:18.864380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.863922    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"
Oct 31 01:50:18.864380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.863946    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn" event={"ID":"46a08ad4-f3cb-421f-963b-5e275efeb8c6","Type":"ContainerDied","Data":"0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726"}
Oct 31 01:50:18.864380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.864006    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn" event={"ID":"46a08ad4-f3cb-421f-963b-5e275efeb8c6","Type":"ContainerDied","Data":"54292973d25b7e5466c81a9c3eb2b8a24f946bd9b3603debb0c8bc845809a4b8"}
Oct 31 01:50:18.864380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.864051    5293 scope.go:122] "RemoveContainer" containerID="0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726"
Oct 31 01:50:18.865719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.865694    5293 generic.go:356] "Generic (PLEG): container finished" podID="c5ac0383-2e25-4ef5-bae0-5372398f534e" containerID="ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db" exitCode=0
Oct 31 01:50:18.865875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.865809    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"
Oct 31 01:50:18.865964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.865944    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b" event={"ID":"c5ac0383-2e25-4ef5-bae0-5372398f534e","Type":"ContainerDied","Data":"ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db"}
Oct 31 01:50:18.866055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.865985    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b" event={"ID":"c5ac0383-2e25-4ef5-bae0-5372398f534e","Type":"ContainerDied","Data":"5b07ea36d1309a224ee778c96cba200397579acffcc5a058f72386887386ede2"}
Oct 31 01:50:18.871196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.871137    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbplj\" (UniqueName: \"kubernetes.io/projected/46a08ad4-f3cb-421f-963b-5e275efeb8c6-kube-api-access-jbplj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:18.871196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.871166    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5pzs\" (UniqueName: \"kubernetes.io/projected/c5ac0383-2e25-4ef5-bae0-5372398f534e-kube-api-access-g5pzs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:18.877648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.877541    5293 scope.go:122] "RemoveContainer" containerID="0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726"
Oct 31 01:50:18.877873 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:50:18.877837    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726\": container with ID starting with 0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726 not found: ID does not exist" containerID="0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726"
Oct 31 01:50:18.877971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.877886    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726"} err="failed to get container status \"0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726\": rpc error: code = NotFound desc = could not find container \"0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726\": container with ID starting with 0f39e08bfc34191a74eea3da57e0a5403b4f63b54048b0b4cc92055a19fda726 not found: ID does not exist"
Oct 31 01:50:18.877971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.877910    5293 scope.go:122] "RemoveContainer" containerID="ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db"
Oct 31 01:50:18.888766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.888750    5293 scope.go:122] "RemoveContainer" containerID="ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db"
Oct 31 01:50:18.889081 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:50:18.889030    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db\": container with ID starting with ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db not found: ID does not exist" containerID="ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db"
Oct 31 01:50:18.889161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.889078    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db"} err="failed to get container status \"ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db\": rpc error: code = NotFound desc = could not find container \"ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db\": container with ID starting with ef6c0b08a80112d077a995ec92c1a43b13e8873b79011b2476599187b66ce0db not found: ID does not exist"
Oct 31 01:50:18.893238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.893217    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"]
Oct 31 01:50:18.899295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.899266    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbbaec-7b746747f9-xdm8b"]
Oct 31 01:50:18.912993 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.912967    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"]
Oct 31 01:50:18.919169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:18.919137    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbbaec-785bd5dfb4-c6wrn"]
Oct 31 01:50:20.180280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:20.180250    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a08ad4-f3cb-421f-963b-5e275efeb8c6" path="/var/lib/kubelet/pods/46a08ad4-f3cb-421f-963b-5e275efeb8c6/volumes"
Oct 31 01:50:20.180783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:20.180768    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bedd736-5a32-460b-967a-ef94e208a02d" path="/var/lib/kubelet/pods/9bedd736-5a32-460b-967a-ef94e208a02d/volumes"
Oct 31 01:50:20.181201 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:20.181186    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ac0383-2e25-4ef5-bae0-5372398f534e" path="/var/lib/kubelet/pods/c5ac0383-2e25-4ef5-bae0-5372398f534e/volumes"
Oct 31 01:50:25.028939 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:25.028887    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"]
Oct 31 01:50:25.034013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:25.033975    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcaa8ee8-969nx"]
Oct 31 01:50:26.192082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.190778    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3990a2d-4452-4f9b-9b13-d19e6d9d8e85" path="/var/lib/kubelet/pods/f3990a2d-4452-4f9b-9b13-d19e6d9d8e85/volumes"
Oct 31 01:50:26.338428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.338389    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"]
Oct 31 01:50:26.339195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339173    5293 cpu_manager.go:414] "containerMap: removing container" podUID="46a08ad4-f3cb-421f-963b-5e275efeb8c6" containerName="payload-processing"
Oct 31 01:50:26.339195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339193    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a08ad4-f3cb-421f-963b-5e275efeb8c6" containerName="payload-processing"
Oct 31 01:50:26.339337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339209    5293 cpu_manager.go:414] "containerMap: removing container" podUID="80424e2c-21e7-430d-9f3b-9449b045fef2" containerName="revoke-keys"
Oct 31 01:50:26.339337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339215    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80424e2c-21e7-430d-9f3b-9449b045fef2" containerName="revoke-keys"
Oct 31 01:50:26.339337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339245    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c5ac0383-2e25-4ef5-bae0-5372398f534e" containerName="payload-pre-processing"
Oct 31 01:50:26.339337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339251    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ac0383-2e25-4ef5-bae0-5372398f534e" containerName="payload-pre-processing"
Oct 31 01:50:26.339482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339369    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a08ad4-f3cb-421f-963b-5e275efeb8c6" containerName="payload-processing"
Oct 31 01:50:26.339482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339380    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="80424e2c-21e7-430d-9f3b-9449b045fef2" containerName="revoke-keys"
Oct 31 01:50:26.339482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339387    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="546bcf87-5caf-4136-8a1f-d6719ba577cb" containerName="istio-proxy"
Oct 31 01:50:26.339482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.339396    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ac0383-2e25-4ef5-bae0-5372398f534e" containerName="payload-pre-processing"
Oct 31 01:50:26.344671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.344649    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.370333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.370196    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"]
Oct 31 01:50:26.395821 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.395789    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a779b3-openshift-default-dockercfg-nkdjw\""
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416317    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416423    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwpb\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-kube-api-access-7kwpb\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416463    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416492    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b865fc7-6a25-467c-b154-45b71658520c-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416572    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416729    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416812    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416860    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b865fc7-6a25-467c-b154-45b71658520c-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.431068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.416913    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.522115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.518210    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwpb\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-kube-api-access-7kwpb\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.547064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.542851    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.639203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.639166    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b865fc7-6a25-467c-b154-45b71658520c-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.639537 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.568586    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.639737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.639714    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.640028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.640008    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.640263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.640238    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwpb\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-kube-api-access-7kwpb\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.640329 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.640242    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.640387 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.640338    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b865fc7-6a25-467c-b154-45b71658520c-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.640437 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.640402    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.640498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.640446    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.640868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.640846    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.641289 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.641238    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.641720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.641675    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.664066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.653761    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b865fc7-6a25-467c-b154-45b71658520c-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.679072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.668267    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b865fc7-6a25-467c-b154-45b71658520c-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.699775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.698473    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.728851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.728777    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:26.964166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:26.964125    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:27.050164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:27.050094    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"]
Oct 31 01:50:27.053858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:27.053829    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a43f65-gz9lg"]
Oct 31 01:50:27.347082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:27.347018    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"]
Oct 31 01:50:27.349276 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:50:27.349248    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b865fc7_6a25_467c_b154_45b71658520c.slice/crio-a0244a1bcd071570a125bfa5c0f9d48354f3dc800d297b86fdf2d85af818e1aa WatchSource:0}: Error finding container a0244a1bcd071570a125bfa5c0f9d48354f3dc800d297b86fdf2d85af818e1aa: Status 404 returned error can't find the container with id a0244a1bcd071570a125bfa5c0f9d48354f3dc800d297b86fdf2d85af818e1aa
Oct 31 01:50:27.352739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:27.352701    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:50:27.352859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:27.352788    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:50:27.352859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:27.352833    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:50:27.432169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:27.432145    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"]
Oct 31 01:50:27.448149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:27.448080    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"]
Oct 31 01:50:28.064061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.054116    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"]
Oct 31 01:50:28.112403 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.112364    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.145225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.141522    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-611693-openshift-default-dockercfg-4cdtj\""
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250120    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-certs\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250267    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-credential-socket\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250390    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-envoy\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250560    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-data\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250602    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-token\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250692    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istiod-ca-cert\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250792    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtqg\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-kube-api-access-ngtqg\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250871    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-socket\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.259065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.250932    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-podinfo\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483001    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-socket\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483120    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-podinfo\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483161    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-certs\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483191    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-credential-socket\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483289    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-envoy\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483434    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-data\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483500    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-token\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483585    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istiod-ca-cert\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.483678    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtqg\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-kube-api-access-ngtqg\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.488088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.484452    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-socket\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.490169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.490136    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-data\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.514058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.492576    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-credential-socket\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.540852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.492783    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-certs\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.541289 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.540751    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istiod-ca-cert\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.553112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.548148    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtqg\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-kube-api-access-ngtqg\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.576702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.576615    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-podinfo\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.617091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.601248    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-envoy\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.631760 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.631696    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" podStartSLOduration=2.6316762970000003 podStartE2EDuration="2.631676297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:28.630715857 +0000 UTC m=+1786.579364251" watchObservedRunningTime="2026-10-31 01:50:28.631676297 +0000 UTC m=+1786.580324690"
Oct 31 01:50:28.670755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.644306    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e336afe-c771-4703-8519-63dd8c922a24" path="/var/lib/kubelet/pods/7e336afe-c771-4703-8519-63dd8c922a24/volumes"
Oct 31 01:50:28.670755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.665264    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-token\") pod \"e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") " pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.670755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.644911    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" event={"ID":"1b865fc7-6a25-467c-b154-45b71658520c","Type":"ContainerStarted","Data":"86c8a4e0628769a4bc09c955e834d9843ab7c45be5753ea4bf6b4c30c27d3ccf"}
Oct 31 01:50:28.670755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.665493    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"]
Oct 31 01:50:28.670755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.665536    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" event={"ID":"1b865fc7-6a25-467c-b154-45b71658520c","Type":"ContainerStarted","Data":"a0244a1bcd071570a125bfa5c0f9d48354f3dc800d297b86fdf2d85af818e1aa"}
Oct 31 01:50:28.764439 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.764350    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:28.976568 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.976526    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:28.982351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:28.982321    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:29.180718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.180684    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"]
Oct 31 01:50:29.182679 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:50:29.181384    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a02efa4_83e1_4b47_a649_c3f3eaccb48f.slice/crio-91c5ba048bd514fc1280d663e621209aa3c571c8b9769fa28b54721f8d5888da WatchSource:0}: Error finding container 91c5ba048bd514fc1280d663e621209aa3c571c8b9769fa28b54721f8d5888da: Status 404 returned error can't find the container with id 91c5ba048bd514fc1280d663e621209aa3c571c8b9769fa28b54721f8d5888da
Oct 31 01:50:29.190097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.190057    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:50:29.190194 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.190154    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:50:29.190232 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.190196    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:50:29.605682 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.605588    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" event={"ID":"0a02efa4-83e1-4b47-a649-c3f3eaccb48f","Type":"ContainerStarted","Data":"94555fa21e87680ce571eeb91cc4e3cb65bca72d71e2877b84f9ed98d71f9be5"}
Oct 31 01:50:29.605682 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.605652    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" event={"ID":"0a02efa4-83e1-4b47-a649-c3f3eaccb48f","Type":"ContainerStarted","Data":"91c5ba048bd514fc1280d663e621209aa3c571c8b9769fa28b54721f8d5888da"}
Oct 31 01:50:29.606239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.606162    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:29.607175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.607152    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:50:29.633020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.632970    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podStartSLOduration=1.6329527160000001 podStartE2EDuration="1.632952716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:29.626708208 +0000 UTC m=+1787.575356601" watchObservedRunningTime="2026-10-31 01:50:29.632952716 +0000 UTC m=+1787.581601108"
Oct 31 01:50:29.769273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:29.769236    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:30.771884 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:30.771855    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:31.624369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:31.624331    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:31.625443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:31.625418    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:50:32.469118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.466583    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h" podUID="04baaad8-eb5e-4421-9fd8-9e77f229b5af" containerName="payload-processing" containerID="cri-o://4357ea3d00c8e8cd812e65c83e3f87aae8786aeab60aaccbd5d64a521ec6a011" gracePeriod=25
Oct 31 01:50:32.483158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.483107    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh" podUID="0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2" containerName="payload-pre-processing" containerID="cri-o://54fd4b480c44ee14376cb537e029415add21e9451a57f972dc0bf569ca4b966b" gracePeriod=25
Oct 31 01:50:32.647022 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.646985    5293 generic.go:356] "Generic (PLEG): container finished" podID="0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2" containerID="54fd4b480c44ee14376cb537e029415add21e9451a57f972dc0bf569ca4b966b" exitCode=0
Oct 31 01:50:32.647309 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.647278    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh" event={"ID":"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2","Type":"ContainerDied","Data":"54fd4b480c44ee14376cb537e029415add21e9451a57f972dc0bf569ca4b966b"}
Oct 31 01:50:32.662254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.659986    5293 generic.go:356] "Generic (PLEG): container finished" podID="04baaad8-eb5e-4421-9fd8-9e77f229b5af" containerID="4357ea3d00c8e8cd812e65c83e3f87aae8786aeab60aaccbd5d64a521ec6a011" exitCode=0
Oct 31 01:50:32.662254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.661149    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h" event={"ID":"04baaad8-eb5e-4421-9fd8-9e77f229b5af","Type":"ContainerDied","Data":"4357ea3d00c8e8cd812e65c83e3f87aae8786aeab60aaccbd5d64a521ec6a011"}
Oct 31 01:50:32.773831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.773729    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"]
Oct 31 01:50:32.790948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.790915    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"]
Oct 31 01:50:32.791105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.791067    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:32.869069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.851359    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a779b3\""
Oct 31 01:50:32.869069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.851514    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a779b3-dockercfg-sgtdc\""
Oct 31 01:50:32.934065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.927652    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/2f6490bd-3a92-467b-9c0e-346efcc86a5a-kube-api-access-nv84g\") pod \"payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk\" (UID: \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:32.934065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.927903    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6490bd-3a92-467b-9c0e-346efcc86a5a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk\" (UID: \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:32.990364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:32.990319    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:33.042651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.040822    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-kube-api-access-frwbd\" (UniqueName: \"kubernetes.io/projected/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-kube-api-access-frwbd\") pod \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\" (UID: \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\") "
Oct 31 01:50:33.042651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.041006    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-plugins-config-volume\") pod \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\" (UID: \"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2\") "
Oct 31 01:50:33.042651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.041577    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6490bd-3a92-467b-9c0e-346efcc86a5a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk\" (UID: \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:33.042651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.041889    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/2f6490bd-3a92-467b-9c0e-346efcc86a5a-kube-api-access-nv84g\") pod \"payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk\" (UID: \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:33.055304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.053141    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6490bd-3a92-467b-9c0e-346efcc86a5a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk\" (UID: \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:33.055304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.055246    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-plugins-config-volume" pod "0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2" (UID: "0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:33.060070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.056980    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-kube-api-access-frwbd" pod "0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2" (UID: "0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2"). InnerVolumeSpecName "kube-api-access-frwbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:33.069121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.067892    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:33.104378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.103471    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/2f6490bd-3a92-467b-9c0e-346efcc86a5a-kube-api-access-nv84g\") pod \"payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk\" (UID: \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:33.284904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.249839    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04baaad8-eb5e-4421-9fd8-9e77f229b5af-kube-api-access-fg4vn\" (UniqueName: \"kubernetes.io/projected/04baaad8-eb5e-4421-9fd8-9e77f229b5af-kube-api-access-fg4vn\") pod \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\" (UID: \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\") "
Oct 31 01:50:33.284904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.249955    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04baaad8-eb5e-4421-9fd8-9e77f229b5af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04baaad8-eb5e-4421-9fd8-9e77f229b5af-plugins-config-volume\") pod \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\" (UID: \"04baaad8-eb5e-4421-9fd8-9e77f229b5af\") "
Oct 31 01:50:33.284904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.250736    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwbd\" (UniqueName: \"kubernetes.io/projected/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-kube-api-access-frwbd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:33.284904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.250759    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:33.284904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.255909    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04baaad8-eb5e-4421-9fd8-9e77f229b5af-kube-api-access-fg4vn" pod "04baaad8-eb5e-4421-9fd8-9e77f229b5af" (UID: "04baaad8-eb5e-4421-9fd8-9e77f229b5af"). InnerVolumeSpecName "kube-api-access-fg4vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:33.284904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.256282    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04baaad8-eb5e-4421-9fd8-9e77f229b5af-plugins-config-volume" pod "04baaad8-eb5e-4421-9fd8-9e77f229b5af" (UID: "04baaad8-eb5e-4421-9fd8-9e77f229b5af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:33.352752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.352716    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg4vn\" (UniqueName: \"kubernetes.io/projected/04baaad8-eb5e-4421-9fd8-9e77f229b5af-kube-api-access-fg4vn\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:33.352752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.352748    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04baaad8-eb5e-4421-9fd8-9e77f229b5af-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:33.361116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.361091    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:33.649926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.649901    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"]
Oct 31 01:50:33.652475 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:50:33.652447    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6490bd_3a92_467b_9c0e_346efcc86a5a.slice/crio-2e1a97455ffae01aa6dc5176a138d2b1d654e59b9242673d75bebf70d57fbb66 WatchSource:0}: Error finding container 2e1a97455ffae01aa6dc5176a138d2b1d654e59b9242673d75bebf70d57fbb66: Status 404 returned error can't find the container with id 2e1a97455ffae01aa6dc5176a138d2b1d654e59b9242673d75bebf70d57fbb66
Oct 31 01:50:33.785251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.785215    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh" event={"ID":"0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2","Type":"ContainerDied","Data":"91e22a1c93dbe75434c36e2a6e7e2c0c27743736dbf6c1098e36d7eb9a1458d7"}
Oct 31 01:50:33.785251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.785233    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"
Oct 31 01:50:33.785482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.785273    5293 scope.go:122] "RemoveContainer" containerID="54fd4b480c44ee14376cb537e029415add21e9451a57f972dc0bf569ca4b966b"
Oct 31 01:50:33.787967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.787939    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk" event={"ID":"2f6490bd-3a92-467b-9c0e-346efcc86a5a","Type":"ContainerStarted","Data":"2e1a97455ffae01aa6dc5176a138d2b1d654e59b9242673d75bebf70d57fbb66"}
Oct 31 01:50:33.791454 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.791430    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"
Oct 31 01:50:33.795448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.795424    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h" event={"ID":"04baaad8-eb5e-4421-9fd8-9e77f229b5af","Type":"ContainerDied","Data":"35ea9cd832bcff31d9334392e8fc5eb57c2a344c0f104e286b70cf74dddc921a"}
Oct 31 01:50:33.824261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.824235    5293 scope.go:122] "RemoveContainer" containerID="4357ea3d00c8e8cd812e65c83e3f87aae8786aeab60aaccbd5d64a521ec6a011"
Oct 31 01:50:33.856018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.855990    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"]
Oct 31 01:50:33.860716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.860694    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-74cfd6d8b9-f2b7h"]
Oct 31 01:50:33.887842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.887758    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"]
Oct 31 01:50:33.894380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.894357    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-68b4f57f87-472dh"]
Oct 31 01:50:33.965899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.965857    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"]
Oct 31 01:50:33.966569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.966555    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2" containerName="payload-pre-processing"
Oct 31 01:50:33.966613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.966572    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2" containerName="payload-pre-processing"
Oct 31 01:50:33.966613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.966609    5293 cpu_manager.go:414] "containerMap: removing container" podUID="04baaad8-eb5e-4421-9fd8-9e77f229b5af" containerName="payload-processing"
Oct 31 01:50:33.966681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.966614    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04baaad8-eb5e-4421-9fd8-9e77f229b5af" containerName="payload-processing"
Oct 31 01:50:33.966720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.966709    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="04baaad8-eb5e-4421-9fd8-9e77f229b5af" containerName="payload-processing"
Oct 31 01:50:33.966753 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.966721    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2" containerName="payload-pre-processing"
Oct 31 01:50:33.970475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.970453    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:33.987618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:33.987597    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-611693\""
Oct 31 01:50:34.000515 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.000489    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"]
Oct 31 01:50:34.004859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.004842    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:34.049470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.049429    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-611693-dockercfg-xrvh6\""
Oct 31 01:50:34.053902 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.053821    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"]
Oct 31 01:50:34.073509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.073480    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c911e715-4526-4f9a-9a31-68a9b2f38cee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn\" (UID: \"c911e715-4526-4f9a-9a31-68a9b2f38cee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:34.073674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.073601    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm\" (UID: \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:34.073737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.073694    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxrx\" (UniqueName: \"kubernetes.io/projected/c911e715-4526-4f9a-9a31-68a9b2f38cee-kube-api-access-zrxrx\") pod \"payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn\" (UID: \"c911e715-4526-4f9a-9a31-68a9b2f38cee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:34.073790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.073751    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lmt\" (UniqueName: \"kubernetes.io/projected/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-kube-api-access-n4lmt\") pod \"payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm\" (UID: \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:34.091267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.091226    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"]
Oct 31 01:50:34.185063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.175495    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm\" (UID: \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:34.185063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.175659    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxrx\" (UniqueName: \"kubernetes.io/projected/c911e715-4526-4f9a-9a31-68a9b2f38cee-kube-api-access-zrxrx\") pod \"payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn\" (UID: \"c911e715-4526-4f9a-9a31-68a9b2f38cee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:34.185063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.175747    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lmt\" (UniqueName: \"kubernetes.io/projected/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-kube-api-access-n4lmt\") pod \"payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm\" (UID: \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:34.185063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.175922    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c911e715-4526-4f9a-9a31-68a9b2f38cee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn\" (UID: \"c911e715-4526-4f9a-9a31-68a9b2f38cee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:34.185063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.177124    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c911e715-4526-4f9a-9a31-68a9b2f38cee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn\" (UID: \"c911e715-4526-4f9a-9a31-68a9b2f38cee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:34.236073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.233344    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm\" (UID: \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:34.236073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.233885    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxrx\" (UniqueName: \"kubernetes.io/projected/c911e715-4526-4f9a-9a31-68a9b2f38cee-kube-api-access-zrxrx\") pod \"payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn\" (UID: \"c911e715-4526-4f9a-9a31-68a9b2f38cee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:34.236313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.236214    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lmt\" (UniqueName: \"kubernetes.io/projected/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-kube-api-access-n4lmt\") pod \"payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm\" (UID: \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:34.287345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.287310    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:34.410473 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.405430    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:34.512325 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.512290    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04baaad8-eb5e-4421-9fd8-9e77f229b5af" path="/var/lib/kubelet/pods/04baaad8-eb5e-4421-9fd8-9e77f229b5af/volumes"
Oct 31 01:50:34.528883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.528847    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2" path="/var/lib/kubelet/pods/0c3b0146-a7c4-4dc3-b19c-4c28a9b714f2/volumes"
Oct 31 01:50:34.836114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.835950    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk" event={"ID":"2f6490bd-3a92-467b-9c0e-346efcc86a5a","Type":"ContainerStarted","Data":"8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883"}
Oct 31 01:50:34.872065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.837294    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:34.882064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.874475    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk" podStartSLOduration=2.87445243 podStartE2EDuration="2.87445243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:34.867401597 +0000 UTC m=+1792.816049990" watchObservedRunningTime="2026-10-31 01:50:34.87445243 +0000 UTC m=+1792.823100815"
Oct 31 01:50:34.905348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.902882    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"]
Oct 31 01:50:34.920776 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:34.920754    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"]
Oct 31 01:50:35.851304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:35.851245    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm" event={"ID":"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8","Type":"ContainerStarted","Data":"a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3"}
Oct 31 01:50:35.851782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:35.851326    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:35.851782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:35.851355    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm" event={"ID":"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8","Type":"ContainerStarted","Data":"77e095db4d94529f8e06fb330ba33db95637af706fc13756e62d69bb558f5769"}
Oct 31 01:50:35.856106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:35.856012    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn" event={"ID":"c911e715-4526-4f9a-9a31-68a9b2f38cee","Type":"ContainerStarted","Data":"2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074"}
Oct 31 01:50:35.856106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:35.856086    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn" event={"ID":"c911e715-4526-4f9a-9a31-68a9b2f38cee","Type":"ContainerStarted","Data":"ba642ab8fe7123bc4aae7503b51bf251429643cd4b50705b18bba717fb61e6e8"}
Oct 31 01:50:35.856278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:35.856207    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:35.869569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:35.869516    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm" podStartSLOduration=2.869498476 podStartE2EDuration="2.869498476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:35.867886225 +0000 UTC m=+1793.816534617" watchObservedRunningTime="2026-10-31 01:50:35.869498476 +0000 UTC m=+1793.818146869"
Oct 31 01:50:35.886809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:35.886762    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn" podStartSLOduration=2.886748886 podStartE2EDuration="2.886748886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:35.884436594 +0000 UTC m=+1793.833084983" watchObservedRunningTime="2026-10-31 01:50:35.886748886 +0000 UTC m=+1793.835397278"
Oct 31 01:50:39.808052 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:39.804962    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"]
Oct 31 01:50:39.834065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:39.822771    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-676801-flbcj"]
Oct 31 01:50:40.177607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:40.177577    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80424e2c-21e7-430d-9f3b-9449b045fef2" path="/var/lib/kubelet/pods/80424e2c-21e7-430d-9f3b-9449b045fef2/volumes"
Oct 31 01:50:40.534675 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:40.533497    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"]
Oct 31 01:50:40.534675 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:40.533750    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy" containerID="cri-o://26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291" gracePeriod=30
Oct 31 01:50:40.562061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:40.561563    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"]
Oct 31 01:50:40.562061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:40.561797    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy" containerID="cri-o://3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434" gracePeriod=30
Oct 31 01:50:41.834094 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:50:41.834001    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:50:43.913173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:43.913138    5293 scope.go:122] "RemoveContainer" containerID="3aa82a556f25598da65c83b972dd70a42396c5daf46d18416f6cae0768e3ed4c"
Oct 31 01:50:44.002053 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.002004    5293 scope.go:122] "RemoveContainer" containerID="9c10d58fdb576f1354bda893a0726f1098145f367faf5b9377f87937232fcf24"
Oct 31 01:50:44.027997 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.027959    5293 scope.go:122] "RemoveContainer" containerID="d5932834b2b4f2dc96db442fb2f2494cbef694c0ad9a1fc6b567711ec4fbae5b"
Oct 31 01:50:44.044227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.044200    5293 scope.go:122] "RemoveContainer" containerID="e37f746c588f5fb59d7fb8507b1e8f537c48f3b046c327671b1620ec20059976"
Oct 31 01:50:44.057092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.057069    5293 scope.go:122] "RemoveContainer" containerID="0c7468e2537178f18dbf952bc3dcda86026d22f3096b401aa405009e27d38db7"
Oct 31 01:50:44.069448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.069424    5293 scope.go:122] "RemoveContainer" containerID="18d4e5c8b9a592436ff597465f526b2debb63df9716ecffafcc095c0dc5df23a"
Oct 31 01:50:44.079564 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.079545    5293 scope.go:122] "RemoveContainer" containerID="59804c338a12beb1f71df8e6ae49ebdcb5438e00bcde852bbcf838f6fb60364d"
Oct 31 01:50:44.089447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.089425    5293 scope.go:122] "RemoveContainer" containerID="98c48f172cc3f884cac8bab05374a3f5f625d830bed053e6175aebe6b8b9a078"
Oct 31 01:50:44.098749 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.098727    5293 scope.go:122] "RemoveContainer" containerID="044208b6e46f48342da2d020b2c59375bb7c882d769cb2a1ab430eb207508318"
Oct 31 01:50:44.108434 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:44.108415    5293 scope.go:122] "RemoveContainer" containerID="dccb29472cd39bb4bf4586dea2b105c911b72d4870ba13cbe41df9fce018a6d2"
Oct 31 01:50:45.882584 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.882560    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:50:45.927387 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.927361    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-kube-api-access-8rjp8\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-kube-api-access-8rjp8\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.927528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.927449    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istiod-ca-cert\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.927603 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.927539    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-credential-socket\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.927873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.927654    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-socket\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.927978 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.927941    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-credential-socket" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:45.928257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.928104    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-data\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.928257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.928210    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-socket" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:45.928257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.928220    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istiod-ca-cert" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:45.928563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.928533    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-data" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:45.928641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.928563    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-token\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-token\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.928641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.928630    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-podinfo\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.928801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.928673    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-certs\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.928801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.928740    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-envoy\") pod \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\" (UID: \"248a307a-a43d-4d2c-aade-59fb8e0efc2a\") "
Oct 31 01:50:45.929305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.929243    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-certs" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:45.929795 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.929776    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:45.929917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.929799    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:45.929917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.929813    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:45.929917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.929827    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:45.929917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.929839    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:45.937208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.937147    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-kube-api-access-8rjp8" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "kube-api-access-8rjp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:45.937208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.937184    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-token" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:45.937475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.937447    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-podinfo" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:45.938009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.937918    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-envoy" pod "248a307a-a43d-4d2c-aade-59fb8e0efc2a" (UID: "248a307a-a43d-4d2c-aade-59fb8e0efc2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:45.971987 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:45.971953    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:50:46.031418 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031391    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-kube-api-access-c99l6\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-kube-api-access-c99l6\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.031615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031446    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-token\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-token\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.031615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031485    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-socket\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.031615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031549    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-podinfo\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.031778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031641    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-data\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.031778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031760    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-credential-socket\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.031891 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031871    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f54a9a0-1e40-4213-a73e-554d493f9313-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f54a9a0-1e40-4213-a73e-554d493f9313-istiod-ca-cert\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.031967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031934    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-envoy\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.032021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.031964    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-certs\") pod \"6f54a9a0-1e40-4213-a73e-554d493f9313\" (UID: \"6f54a9a0-1e40-4213-a73e-554d493f9313\") "
Oct 31 01:50:46.033486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.032808    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.033486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.032825    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-data" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:46.033486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.032837    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.033486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.032927    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248a307a-a43d-4d2c-aade-59fb8e0efc2a-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.033486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.032946    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjp8\" (UniqueName: \"kubernetes.io/projected/248a307a-a43d-4d2c-aade-59fb8e0efc2a-kube-api-access-8rjp8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.033486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.033128    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-certs" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:46.033486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.033254    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f54a9a0-1e40-4213-a73e-554d493f9313-istiod-ca-cert" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:46.034208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.034171    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-credential-socket" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:46.034495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.034472    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-socket" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:46.036812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.036786    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-kube-api-access-c99l6" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "kube-api-access-c99l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:46.043442 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.043370    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-podinfo" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:46.044238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.044214    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-envoy" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:46.046598 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.046576    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-token" pod "6f54a9a0-1e40-4213-a73e-554d493f9313" (UID: "6f54a9a0-1e40-4213-a73e-554d493f9313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:46.086836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.086756    5293 generic.go:356] "Generic (PLEG): container finished" podID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerID="3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434" exitCode=0
Oct 31 01:50:46.086836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.086829    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"
Oct 31 01:50:46.087106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.086850    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" event={"ID":"6f54a9a0-1e40-4213-a73e-554d493f9313","Type":"ContainerDied","Data":"3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434"}
Oct 31 01:50:46.087106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.086904    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl" event={"ID":"6f54a9a0-1e40-4213-a73e-554d493f9313","Type":"ContainerDied","Data":"906c0cdb160e0670e3ca8109ff7ebbaf86de8e54d3d7d77a0c74abdbfbdb45ad"}
Oct 31 01:50:46.087106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.086935    5293 scope.go:122] "RemoveContainer" containerID="3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434"
Oct 31 01:50:46.088971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.088949    5293 generic.go:356] "Generic (PLEG): container finished" podID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerID="26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291" exitCode=0
Oct 31 01:50:46.089105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.089001    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"
Oct 31 01:50:46.089105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.089012    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" event={"ID":"248a307a-a43d-4d2c-aade-59fb8e0efc2a","Type":"ContainerDied","Data":"26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291"}
Oct 31 01:50:46.089105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.089086    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf" event={"ID":"248a307a-a43d-4d2c-aade-59fb8e0efc2a","Type":"ContainerDied","Data":"ccc2749ebf3fb5c2827c4558e70cda1f58374091ffac1153ff21bf8722eeb263"}
Oct 31 01:50:46.099420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.099382    5293 scope.go:122] "RemoveContainer" containerID="3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434"
Oct 31 01:50:46.099739 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:50:46.099716    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434\": container with ID starting with 3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434 not found: ID does not exist" containerID="3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434"
Oct 31 01:50:46.099828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.099750    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434"} err="failed to get container status \"3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434\": rpc error: code = NotFound desc = could not find container \"3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434\": container with ID starting with 3be68b450bc9702351550b2d35caff7bf071440e28c4e30d7ec99e48e2531434 not found: ID does not exist"
Oct 31 01:50:46.099828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.099774    5293 scope.go:122] "RemoveContainer" containerID="26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291"
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138850    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138895    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138910    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f54a9a0-1e40-4213-a73e-554d493f9313-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138924    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138937    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138951    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99l6\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-kube-api-access-c99l6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138965    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138977    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f54a9a0-1e40-4213-a73e-554d493f9313-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.150086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.138990    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f54a9a0-1e40-4213-a73e-554d493f9313-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:46.169062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.166293    5293 scope.go:122] "RemoveContainer" containerID="26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291"
Oct 31 01:50:46.170655 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:50:46.170525    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291\": container with ID starting with 26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291 not found: ID does not exist" containerID="26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291"
Oct 31 01:50:46.170655 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.170573    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291"} err="failed to get container status \"26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291\": rpc error: code = NotFound desc = could not find container \"26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291\": container with ID starting with 26825ffde8a7bc40b032cbe4178ae8acf35b3d7fe4e0f4f770d6ea1397732291 not found: ID does not exist"
Oct 31 01:50:46.248198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.248141    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"]
Oct 31 01:50:46.282063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.281099    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-q47tl"]
Oct 31 01:50:46.324078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.314104    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"]
Oct 31 01:50:46.359417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.358860    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-nd2rf"]
Oct 31 01:50:46.881638 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.881605    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:50:46.881953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.881926    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:50:46.882804 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:46.882788    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:50:47.636620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:47.636586    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"]
Oct 31 01:50:47.636884 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:47.636857    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" containerID="cri-o://c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3" gracePeriod=30
Oct 31 01:50:48.176411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.176375    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" path="/var/lib/kubelet/pods/248a307a-a43d-4d2c-aade-59fb8e0efc2a/volumes"
Oct 31 01:50:48.177091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.177057    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" path="/var/lib/kubelet/pods/6f54a9a0-1e40-4213-a73e-554d493f9313/volumes"
Oct 31 01:50:48.228243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.228214    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"]
Oct 31 01:50:48.228875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.228860    5293 cpu_manager.go:414] "containerMap: removing container" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy"
Oct 31 01:50:48.228927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.228877    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy"
Oct 31 01:50:48.228927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.228900    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy"
Oct 31 01:50:48.228927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.228905    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy"
Oct 31 01:50:48.229024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.229015    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="248a307a-a43d-4d2c-aade-59fb8e0efc2a" containerName="istio-proxy"
Oct 31 01:50:48.229080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.229029    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f54a9a0-1e40-4213-a73e-554d493f9313" containerName="istio-proxy"
Oct 31 01:50:48.233979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.233960    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.237916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.237882    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:50:48.238095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.237928    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:50:48.245363 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.245340    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"]
Oct 31 01:50:48.268395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.268363    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.268678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.268647    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtjj\" (UniqueName: \"kubernetes.io/projected/313e19da-61b5-41ba-b65b-ad985a697ed1-kube-api-access-5jtjj\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.268835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.268812    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.280276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.280252    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:50:48.280445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.280430    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:50:48.371650 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.370704    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtjj\" (UniqueName: \"kubernetes.io/projected/313e19da-61b5-41ba-b65b-ad985a697ed1-kube-api-access-5jtjj\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.371851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.371689    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.371851 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.371828    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.372532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.372505    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.375112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.375084    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.387084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.387010    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtjj\" (UniqueName: \"kubernetes.io/projected/313e19da-61b5-41ba-b65b-ad985a697ed1-kube-api-access-5jtjj\") pod \"maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.557128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.553866    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:48.868480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:48.868452    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"]
Oct 31 01:50:48.869151 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:50:48.869120    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313e19da_61b5_41ba_b65b_ad985a697ed1.slice/crio-d7c12b601c4f758acb1090b7909303b0dc17c90df1e5bce67ddb51acac41d3d6 WatchSource:0}: Error finding container d7c12b601c4f758acb1090b7909303b0dc17c90df1e5bce67ddb51acac41d3d6: Status 404 returned error can't find the container with id d7c12b601c4f758acb1090b7909303b0dc17c90df1e5bce67ddb51acac41d3d6
Oct 31 01:50:49.120826 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:49.120733    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79" event={"ID":"313e19da-61b5-41ba-b65b-ad985a697ed1","Type":"ContainerStarted","Data":"d7c12b601c4f758acb1090b7909303b0dc17c90df1e5bce67ddb51acac41d3d6"}
Oct 31 01:50:50.180020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:50.179944    5293 generic.go:356] "Generic (PLEG): container finished" podID="313e19da-61b5-41ba-b65b-ad985a697ed1" containerID="77aa574571ae3d9b68db331ffb4804680adff6bd17b04555f1ad78aa95fdd8d1" exitCode=0
Oct 31 01:50:50.180450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:50.180019    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79" event={"ID":"313e19da-61b5-41ba-b65b-ad985a697ed1","Type":"ContainerDied","Data":"77aa574571ae3d9b68db331ffb4804680adff6bd17b04555f1ad78aa95fdd8d1"}
Oct 31 01:50:51.366769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.366739    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:51.494803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.494725    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313e19da-61b5-41ba-b65b-ad985a697ed1-kube-api-access-5jtjj\" (UniqueName: \"kubernetes.io/projected/313e19da-61b5-41ba-b65b-ad985a697ed1-kube-api-access-5jtjj\") pod \"313e19da-61b5-41ba-b65b-ad985a697ed1\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") "
Oct 31 01:50:51.494986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.494813    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-openshift-ca\") pod \"313e19da-61b5-41ba-b65b-ad985a697ed1\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") "
Oct 31 01:50:51.495098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.494989    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-xks-ca\") pod \"313e19da-61b5-41ba-b65b-ad985a697ed1\" (UID: \"313e19da-61b5-41ba-b65b-ad985a697ed1\") "
Oct 31 01:50:51.495259 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.495235    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-openshift-ca" pod "313e19da-61b5-41ba-b65b-ad985a697ed1" (UID: "313e19da-61b5-41ba-b65b-ad985a697ed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:51.495689 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.495627    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:51.497127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.497103    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313e19da-61b5-41ba-b65b-ad985a697ed1-kube-api-access-5jtjj" pod "313e19da-61b5-41ba-b65b-ad985a697ed1" (UID: "313e19da-61b5-41ba-b65b-ad985a697ed1"). InnerVolumeSpecName "kube-api-access-5jtjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:51.497225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.497164    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-xks-ca" pod "313e19da-61b5-41ba-b65b-ad985a697ed1" (UID: "313e19da-61b5-41ba-b65b-ad985a697ed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:51.615069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.614969    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/313e19da-61b5-41ba-b65b-ad985a697ed1-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:51.615069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:51.615016    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jtjj\" (UniqueName: \"kubernetes.io/projected/313e19da-61b5-41ba-b65b-ad985a697ed1-kube-api-access-5jtjj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:52.201512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.201477    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79" event={"ID":"313e19da-61b5-41ba-b65b-ad985a697ed1","Type":"ContainerDied","Data":"d7c12b601c4f758acb1090b7909303b0dc17c90df1e5bce67ddb51acac41d3d6"}
Oct 31 01:50:52.201647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.201535    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7c12b601c4f758acb1090b7909303b0dc17c90df1e5bce67ddb51acac41d3d6"
Oct 31 01:50:52.201647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.201490    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"
Oct 31 01:50:52.302544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.302498    5293 patch_prober.go:29] interesting pod/e2e-api-a-676801-openshift-default-d68754d77-nljqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:50:52.302691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.302562    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:50:52.951974 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.951949    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:50:52.972605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.972575    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istiod-ca-cert\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.972741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.972617    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-credential-socket\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.972791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.972749    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-envoy\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.972791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.972784    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-podinfo\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.972896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.972869    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-kube-api-access-mbrnv\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-kube-api-access-mbrnv\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.972955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.972925    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-credential-socket" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:52.972955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.972934    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istiod-ca-cert" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:52.973130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973109    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-data\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.973190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973160    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-certs\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.973227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973199    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-token\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-token\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.973288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973237    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-socket\") pod \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\" (UID: \"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3\") "
Oct 31 01:50:52.973485 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973459    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-data" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:52.973561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973513    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-certs" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:52.973561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973523    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-socket" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:52.973937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973918    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:52.974060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973939    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:52.974060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973954    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:52.974060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973966    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:52.974060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.973978    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:52.976384 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.976360    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-kube-api-access-mbrnv" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "kube-api-access-mbrnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:52.976639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.976616    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-envoy" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:52.978420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.978397    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-token" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:52.980666 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:52.980639    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-podinfo" pod "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" (UID: "e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:53.112320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.112271    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbrnv\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-kube-api-access-mbrnv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:53.112320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.112317    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:53.112570 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.112332    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:53.112570 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.112345    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:53.212222 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.212140    5293 generic.go:356] "Generic (PLEG): container finished" podID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerID="c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3" exitCode=0
Oct 31 01:50:53.212222 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.212188    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"
Oct 31 01:50:53.212418 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.212183    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" event={"ID":"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3","Type":"ContainerDied","Data":"c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3"}
Oct 31 01:50:53.212418 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.212310    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb" event={"ID":"e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3","Type":"ContainerDied","Data":"32a521e7b2f186063ab93f327ca9100a58abbe59b8087b1c91694ccdf6c2414b"}
Oct 31 01:50:53.212418 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.212341    5293 scope.go:122] "RemoveContainer" containerID="c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3"
Oct 31 01:50:53.224636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.224615    5293 scope.go:122] "RemoveContainer" containerID="06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3"
Oct 31 01:50:53.236191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.236173    5293 scope.go:122] "RemoveContainer" containerID="c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3"
Oct 31 01:50:53.236557 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:50:53.236538    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3\": container with ID starting with c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3 not found: ID does not exist" containerID="c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3"
Oct 31 01:50:53.236620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.236563    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3"} err="failed to get container status \"c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3\": rpc error: code = NotFound desc = could not find container \"c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3\": container with ID starting with c91c9cf643747ff96eef36a31ed86b9f153e25cba14989b7714e41c5d1581cd3 not found: ID does not exist"
Oct 31 01:50:53.236620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.236581    5293 scope.go:122] "RemoveContainer" containerID="06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3"
Oct 31 01:50:53.236881 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:50:53.236852    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3\": container with ID starting with 06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3 not found: ID does not exist" containerID="06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3"
Oct 31 01:50:53.236974 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.236893    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3"} err="failed to get container status \"06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3\": rpc error: code = NotFound desc = could not find container \"06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3\": container with ID starting with 06c96a4f8f425711ba84b1c2e258d9773fb25de578397e930b48428beb5b77d3 not found: ID does not exist"
Oct 31 01:50:53.237120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.237103    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"]
Oct 31 01:50:53.240852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:53.240832    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-676801-openshift-default-d68754d77-nljqb"]
Oct 31 01:50:54.175560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:54.175524    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" path="/var/lib/kubelet/pods/e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3/volumes"
Oct 31 01:50:55.552788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.552751    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"]
Oct 31 01:50:55.553396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553381    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy"
Oct 31 01:50:55.553443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553397    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy"
Oct 31 01:50:55.553443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553413    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy"
Oct 31 01:50:55.553443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553418    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy"
Oct 31 01:50:55.553534 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553471    5293 cpu_manager.go:414] "containerMap: removing container" podUID="313e19da-61b5-41ba-b65b-ad985a697ed1" containerName="revoke-keys"
Oct 31 01:50:55.553534 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553476    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313e19da-61b5-41ba-b65b-ad985a697ed1" containerName="revoke-keys"
Oct 31 01:50:55.553606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553575    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="313e19da-61b5-41ba-b65b-ad985a697ed1" containerName="revoke-keys"
Oct 31 01:50:55.553606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553585    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy"
Oct 31 01:50:55.553606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.553590    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b7e1ab-0035-42d0-bb0f-41ba91d8a6f3" containerName="istio-proxy"
Oct 31 01:50:55.558952 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.558926    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.572920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.572898    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"]
Oct 31 01:50:55.709236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.709196    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.709236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.709229    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlw2h\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-kube-api-access-dlw2h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.709493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.709346    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46307ab4-5e52-4d6d-a527-9e7756732832-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.709493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.709375    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.709601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.709509    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.709601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.709549    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.709783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.709762    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.810998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.810915    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46307ab4-5e52-4d6d-a527-9e7756732832-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.810998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.810952    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.811343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.811025    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.811343 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.811123    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.811462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.811353    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.811462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.811398    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.811462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.811417    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw2h\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-kube-api-access-dlw2h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.812484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.812449    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.813883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.813846    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.813981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.813845    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.814075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.814017    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.814158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.814141    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46307ab4-5e52-4d6d-a527-9e7756732832-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.822444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.822422    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlw2h\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-kube-api-access-dlw2h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.822603 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.822585    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-chwsm\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:55.873482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:55.873457    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:56.177266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:56.177247    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"]
Oct 31 01:50:56.179552 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:50:56.179520    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46307ab4_5e52_4d6d_a527_9e7756732832.slice/crio-ca66fb40770478a5403f931d0eb791ce64d7861da8a81c92dda02011a8a99bca WatchSource:0}: Error finding container ca66fb40770478a5403f931d0eb791ce64d7861da8a81c92dda02011a8a99bca: Status 404 returned error can't find the container with id ca66fb40770478a5403f931d0eb791ce64d7861da8a81c92dda02011a8a99bca
Oct 31 01:50:56.182426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:56.182391    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:50:56.182530 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:56.182455    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:50:56.323503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:56.323240    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm" event={"ID":"46307ab4-5e52-4d6d-a527-9e7756732832","Type":"ContainerStarted","Data":"ca66fb40770478a5403f931d0eb791ce64d7861da8a81c92dda02011a8a99bca"}
Oct 31 01:50:57.334548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:57.334505    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm" event={"ID":"46307ab4-5e52-4d6d-a527-9e7756732832","Type":"ContainerStarted","Data":"9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d"}
Oct 31 01:50:57.335066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:57.334601    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:50:57.336944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:57.336914    5293 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-chwsm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:50:57.337082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:57.336973    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm" podUID="46307ab4-5e52-4d6d-a527-9e7756732832" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:50:57.359411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:57.359347    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm" podStartSLOduration=2.359329435 podStartE2EDuration="2.359329435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:57.355768099 +0000 UTC m=+1815.304416491" watchObservedRunningTime="2026-10-31 01:50:57.359329435 +0000 UTC m=+1815.307977830"
Oct 31 01:50:58.412069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:50:58.410277    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 01:51:03.735793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:03.733298    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"]
Oct 31 01:51:03.735793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:03.733547    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp" podUID="e0f17b37-c55b-49c2-b5c1-b5c75228ce5a" containerName="payload-processing" containerID="cri-o://758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82" gracePeriod=30
Oct 31 01:51:04.591889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.591853    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:51:04.728729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.713426    5293 generic.go:356] "Generic (PLEG): container finished" podID="e0f17b37-c55b-49c2-b5c1-b5c75228ce5a" containerID="758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82" exitCode=0
Oct 31 01:51:04.729457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.714421    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"
Oct 31 01:51:04.730079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.715001    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp" event={"ID":"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a","Type":"ContainerDied","Data":"758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82"}
Oct 31 01:51:04.730326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.730301    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp" event={"ID":"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a","Type":"ContainerDied","Data":"3422854906ed58727dd22ac1edbc449e3ed24ccb0bd633d296f5e5305c6dc978"}
Oct 31 01:51:04.730478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.730462    5293 scope.go:122] "RemoveContainer" containerID="758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82"
Oct 31 01:51:04.730726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.724181    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-kube-api-access-h5tkj\" (UniqueName: \"kubernetes.io/projected/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-kube-api-access-h5tkj\") pod \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\" (UID: \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\") "
Oct 31 01:51:04.731196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.731176    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-plugins-config-volume\") pod \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\" (UID: \"e0f17b37-c55b-49c2-b5c1-b5c75228ce5a\") "
Oct 31 01:51:04.797070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.796278    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-plugins-config-volume" pod "e0f17b37-c55b-49c2-b5c1-b5c75228ce5a" (UID: "e0f17b37-c55b-49c2-b5c1-b5c75228ce5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:04.837952 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.837921    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:04.848678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.848643    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-kube-api-access-h5tkj" pod "e0f17b37-c55b-49c2-b5c1-b5c75228ce5a" (UID: "e0f17b37-c55b-49c2-b5c1-b5c75228ce5a"). InnerVolumeSpecName "kube-api-access-h5tkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:04.940495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:04.940455    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tkj\" (UniqueName: \"kubernetes.io/projected/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a-kube-api-access-h5tkj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:05.053115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:05.051399    5293 scope.go:122] "RemoveContainer" containerID="758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82"
Oct 31 01:51:05.053115 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:05.051851    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82\": container with ID starting with 758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82 not found: ID does not exist" containerID="758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82"
Oct 31 01:51:05.053115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:05.051885    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82"} err="failed to get container status \"758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82\": rpc error: code = NotFound desc = could not find container \"758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82\": container with ID starting with 758b1b652ba3b3860b186cb8f214e24e09addddb978f1193f25825dd453a0f82 not found: ID does not exist"
Oct 31 01:51:05.077428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:05.077085    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"]
Oct 31 01:51:05.080773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:05.080750    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-6587d56cd4-zd2wp"]
Oct 31 01:51:06.175960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:06.175926    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f17b37-c55b-49c2-b5c1-b5c75228ce5a" path="/var/lib/kubelet/pods/e0f17b37-c55b-49c2-b5c1-b5c75228ce5a/volumes"
Oct 31 01:51:07.049572 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:07.049536    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:51:09.293566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:09.293529    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"]
Oct 31 01:51:09.295763 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:09.295726    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy" containerID="cri-o://0e784058d05de146c9746571856f4fad7c8cefd69d8dc3bff3a33da96bc6ce6c" gracePeriod=30
Oct 31 01:51:09.318334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:09.318300    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"]
Oct 31 01:51:09.318601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:09.318586    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"]
Oct 31 01:51:09.319318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:09.318897    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy" containerID="cri-o://f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29" gracePeriod=30
Oct 31 01:51:09.319739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:09.319711    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" containerID="cri-o://400339eb833ad4dcceac34087b942cd8b9ea5e74f1a4116f34a21c3066ae7906" gracePeriod=30
Oct 31 01:51:11.506641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:11.506240    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:51:11.506641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:11.506326    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:51:11.833112 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:11.828290    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:51:12.613892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:12.613839    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"]
Oct 31 01:51:12.614906 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:12.614840    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy" containerID="cri-o://cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514" gracePeriod=30
Oct 31 01:51:13.320064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.301106    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"]
Oct 31 01:51:13.360064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.359707    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e0f17b37-c55b-49c2-b5c1-b5c75228ce5a" containerName="payload-processing"
Oct 31 01:51:13.360064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.359736    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f17b37-c55b-49c2-b5c1-b5c75228ce5a" containerName="payload-processing"
Oct 31 01:51:13.360064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.359930    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f17b37-c55b-49c2-b5c1-b5c75228ce5a" containerName="payload-processing"
Oct 31 01:51:13.386539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.386281    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"]
Oct 31 01:51:13.386539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.386405    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.560105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.546703    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:51:13.560105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.546922    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:51:13.560105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.547000    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:51:13.560105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.555336    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:51:13.600512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.592467    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzff6\" (UniqueName: \"kubernetes.io/projected/18c50da9-caab-4682-8b9b-50876f0a07ae-kube-api-access-rzff6\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.600512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.592600    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.600512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.592652    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.720883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.720833    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzff6\" (UniqueName: \"kubernetes.io/projected/18c50da9-caab-4682-8b9b-50876f0a07ae-kube-api-access-rzff6\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.721635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.721604    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.721776 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.721761    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.722849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.722812    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.737182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.736393    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.740667 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.740552    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzff6\" (UniqueName: \"kubernetes.io/projected/18c50da9-caab-4682-8b9b-50876f0a07ae-kube-api-access-rzff6\") pod \"maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:13.758623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:13.756994    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:14.594126 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:14.594008    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82_8eb20b7d-f1b3-4773-be3b-335f1271d3a6/payload-processing/0.log"
Oct 31 01:51:14.695129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:14.694966    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"]
Oct 31 01:51:14.758590 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:51:14.718531    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c50da9_caab_4682_8b9b_50876f0a07ae.slice/crio-544ec8bad5c3caf208ce9d8086b04980b14a25abcedab4a51f76b9362784bd7b WatchSource:0}: Error finding container 544ec8bad5c3caf208ce9d8086b04980b14a25abcedab4a51f76b9362784bd7b: Status 404 returned error can't find the container with id 544ec8bad5c3caf208ce9d8086b04980b14a25abcedab4a51f76b9362784bd7b
Oct 31 01:51:14.943421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:14.915536    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82_8eb20b7d-f1b3-4773-be3b-335f1271d3a6/payload-processing/0.log"
Oct 31 01:51:15.392296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.392271    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:51:15.485605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.485567    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-data\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.485803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.485621    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-certs\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.485803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.485724    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-podinfo\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.485803 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.485763    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-kube-api-access-vjhrs\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-kube-api-access-vjhrs\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.485983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.485814    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istiod-ca-cert\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.485983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.485851    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-socket\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.487174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.487145    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-data" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.487791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.487510    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-certs" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.488220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.488198    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istiod-ca-cert" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:15.489281 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.488768    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-socket" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.490076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.490009    5293 generic.go:356] "Generic (PLEG): container finished" podID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerID="400339eb833ad4dcceac34087b942cd8b9ea5e74f1a4116f34a21c3066ae7906" exitCode=0
Oct 31 01:51:15.490215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.490195    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" event={"ID":"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e","Type":"ContainerDied","Data":"400339eb833ad4dcceac34087b942cd8b9ea5e74f1a4116f34a21c3066ae7906"}
Oct 31 01:51:15.491880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.491857    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t" event={"ID":"18c50da9-caab-4682-8b9b-50876f0a07ae","Type":"ContainerStarted","Data":"544ec8bad5c3caf208ce9d8086b04980b14a25abcedab4a51f76b9362784bd7b"}
Oct 31 01:51:15.495354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.495331    5293 generic.go:356] "Generic (PLEG): container finished" podID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerID="0e784058d05de146c9746571856f4fad7c8cefd69d8dc3bff3a33da96bc6ce6c" exitCode=0
Oct 31 01:51:15.495455 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.495403    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" event={"ID":"aff97f77-3447-4ff9-91ea-d7dda29aa919","Type":"ContainerDied","Data":"0e784058d05de146c9746571856f4fad7c8cefd69d8dc3bff3a33da96bc6ce6c"}
Oct 31 01:51:15.525064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.512276    5293 generic.go:356] "Generic (PLEG): container finished" podID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerID="f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29" exitCode=0
Oct 31 01:51:15.525064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.512398    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" event={"ID":"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55","Type":"ContainerDied","Data":"f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29"}
Oct 31 01:51:15.525064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.512447    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr" event={"ID":"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55","Type":"ContainerDied","Data":"d58a07ddfebd90956e79563a9d6daac610f1b10f83ac7e112ac22d01a7251772"}
Oct 31 01:51:15.525064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.512477    5293 scope.go:122] "RemoveContainer" containerID="f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29"
Oct 31 01:51:15.525064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.512650    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"
Oct 31 01:51:15.539398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.538080    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-podinfo" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:15.539398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.538426    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-kube-api-access-vjhrs" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "kube-api-access-vjhrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:15.563779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.563754    5293 scope.go:122] "RemoveContainer" containerID="f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29"
Oct 31 01:51:15.564101 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:15.564065    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29\": container with ID starting with f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29 not found: ID does not exist" containerID="f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29"
Oct 31 01:51:15.564211 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.564115    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29"} err="failed to get container status \"f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29\": rpc error: code = NotFound desc = could not find container \"f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29\": container with ID starting with f6548c67cc290eb9470ff74e1416c3cddc3446a80bf4caa76e8beeacb71a4f29 not found: ID does not exist"
Oct 31 01:51:15.566256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.566234    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:51:15.587281 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587250    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-data\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.587438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587316    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-envoy\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.587438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587381    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-token\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.587438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587414    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-envoy\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.587617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587460    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-credential-socket\") pod \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\" (UID: \"25d5a1ff-ad96-43bd-b3ee-b33072a1cb55\") "
Oct 31 01:51:15.587617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587488    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-certs\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.587617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587518    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-podinfo\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.587617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587582    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-data" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.587617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587602    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-token\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-token\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.587879 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.587789    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-credential-socket" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.588235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588111    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-certs" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.588383 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588361    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.588429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588391    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.588429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588406    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.588429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588422    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.588524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588435    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.588524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588448    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.588524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588464    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.588524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588478    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.588524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.588492    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjhrs\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-kube-api-access-vjhrs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.591573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.591545    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-token" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:15.592545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.592438    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-envoy" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.593646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.593591    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-podinfo" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:15.600606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.600586    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-envoy" pod "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" (UID: "25d5a1ff-ad96-43bd-b3ee-b33072a1cb55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.655322 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.655162    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-token" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:15.657158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.657133    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"]
Oct 31 01:51:15.658489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.658470    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy"
Oct 31 01:51:15.658629 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.658616    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy"
Oct 31 01:51:15.658751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.658741    5293 cpu_manager.go:414] "containerMap: removing container" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy"
Oct 31 01:51:15.658846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.658835    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy"
Oct 31 01:51:15.659198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.659186    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" containerName="istio-proxy"
Oct 31 01:51:15.659322 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.659312    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" containerName="istio-proxy"
Oct 31 01:51:15.664595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.664574    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"]
Oct 31 01:51:15.664749 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.664700    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.689453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.689421    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istiod-ca-cert\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.689687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.689654    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-socket\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.689808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.689786    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istiod-ca-cert" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:15.689940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.689825    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-kube-api-access-lxdt2\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-kube-api-access-lxdt2\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.690061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.690029    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-credential-socket\") pod \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\" (UID: \"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e\") "
Oct 31 01:51:15.690445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.690393    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.690537 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.690483    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-socket" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.690537 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.690525    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-credential-socket" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.690684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.690662    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7wv\" (UniqueName: \"kubernetes.io/projected/b73bc331-24f6-4135-bee9-787caba8cfe3-kube-api-access-nt7wv\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.690828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.690807    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.691291 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.691270    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.691372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.691296    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.691372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.691314    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.691372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.691329    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.691372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.691342    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.691372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.691356    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.691372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.691369    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.691577 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.691383    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.693009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.692991    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-kube-api-access-lxdt2" pod "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" (UID: "b9d9c586-60ff-4ad2-9f84-f9292aa5b28e"). InnerVolumeSpecName "kube-api-access-lxdt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:15.693518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.693501    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:51:15.841587 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.841559    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-token\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-token\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.850550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.850522    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-certs\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.850789 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.850775    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-envoy\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.850896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.850885    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-socket\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.850973 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.850963    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff97f77-3447-4ff9-91ea-d7dda29aa919-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff97f77-3447-4ff9-91ea-d7dda29aa919-istiod-ca-cert\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.851051 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.851027    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-podinfo\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.851325 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.851303    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-kube-api-access-zg5zh\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-kube-api-access-zg5zh\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.851632 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.851611    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-credential-socket\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.852157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.851758    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff97f77-3447-4ff9-91ea-d7dda29aa919-istiod-ca-cert" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:15.852157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.851917    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-data\") pod \"aff97f77-3447-4ff9-91ea-d7dda29aa919\" (UID: \"aff97f77-3447-4ff9-91ea-d7dda29aa919\") "
Oct 31 01:51:15.853294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.853071    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-data" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.853597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.853572    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-socket" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.855629 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.855606    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-kube-api-access-zg5zh" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "kube-api-access-zg5zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:15.855943 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.855926    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-credential-socket" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.856395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.856368    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-envoy" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.856607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.856585    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.856688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.856651    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-certs" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:15.856910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.856884    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.869085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.869013    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877245    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7wv\" (UniqueName: \"kubernetes.io/projected/b73bc331-24f6-4135-bee9-787caba8cfe3-kube-api-access-nt7wv\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877500    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877522    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877537    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877558    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff97f77-3447-4ff9-91ea-d7dda29aa919-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877576    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg5zh\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-kube-api-access-zg5zh\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877591    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877608    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxdt2\" (UniqueName: \"kubernetes.io/projected/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e-kube-api-access-lxdt2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.878241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.877627    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.879397 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.879332    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.889607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.889573    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-token" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:15.890904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.890822    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7wv\" (UniqueName: \"kubernetes.io/projected/b73bc331-24f6-4135-bee9-787caba8cfe3-kube-api-access-nt7wv\") pod \"maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:15.915905 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.915858    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-podinfo" pod "aff97f77-3447-4ff9-91ea-d7dda29aa919" (UID: "aff97f77-3447-4ff9-91ea-d7dda29aa919"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:15.961764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.961701    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"]
Oct 31 01:51:15.967521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.967487    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dlngr"]
Oct 31 01:51:15.979167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.979141    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.979335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.979319    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff97f77-3447-4ff9-91ea-d7dda29aa919-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:15.988752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:15.988722    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:16.177859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.177815    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d5a1ff-ad96-43bd-b3ee-b33072a1cb55" path="/var/lib/kubelet/pods/25d5a1ff-ad96-43bd-b3ee-b33072a1cb55/volumes"
Oct 31 01:51:16.300959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.300907    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"]
Oct 31 01:51:16.301730 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:51:16.301692    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73bc331_24f6_4135_bee9_787caba8cfe3.slice/crio-496c7ae6e5b7d1aadabed252d2abd83438ffe8c2bf21d126d0d23d35abc7e554 WatchSource:0}: Error finding container 496c7ae6e5b7d1aadabed252d2abd83438ffe8c2bf21d126d0d23d35abc7e554: Status 404 returned error can't find the container with id 496c7ae6e5b7d1aadabed252d2abd83438ffe8c2bf21d126d0d23d35abc7e554
Oct 31 01:51:16.593171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.593014    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn" event={"ID":"b73bc331-24f6-4135-bee9-787caba8cfe3","Type":"ContainerStarted","Data":"496c7ae6e5b7d1aadabed252d2abd83438ffe8c2bf21d126d0d23d35abc7e554"}
Oct 31 01:51:16.596335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.596307    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"
Oct 31 01:51:16.596335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.596323    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj" event={"ID":"b9d9c586-60ff-4ad2-9f84-f9292aa5b28e","Type":"ContainerDied","Data":"7c38805e8adb88a797bb2ef0901696df9b1d191b243795d56d86613ab569c3f0"}
Oct 31 01:51:16.596595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.596399    5293 scope.go:122] "RemoveContainer" containerID="400339eb833ad4dcceac34087b942cd8b9ea5e74f1a4116f34a21c3066ae7906"
Oct 31 01:51:16.608791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.608747    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t" event={"ID":"18c50da9-caab-4682-8b9b-50876f0a07ae","Type":"ContainerStarted","Data":"75f9a3ab7a0a045d3b00d0175315e1b2a730acc4e5b43f8a40fc0c16c38d5c59"}
Oct 31 01:51:16.613122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.613092    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl" event={"ID":"aff97f77-3447-4ff9-91ea-d7dda29aa919","Type":"ContainerDied","Data":"e175f075c14aaca41a5bc969c730c216a7cb7666236168dd006dab113bea5453"}
Oct 31 01:51:16.613307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.613220    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"
Oct 31 01:51:16.626494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.626472    5293 scope.go:122] "RemoveContainer" containerID="0e784058d05de146c9746571856f4fad7c8cefd69d8dc3bff3a33da96bc6ce6c"
Oct 31 01:51:16.637635 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.637578    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t" podStartSLOduration=3.63756359 podStartE2EDuration="3.63756359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:16.627829505 +0000 UTC m=+1834.576477891" watchObservedRunningTime="2026-10-31 01:51:16.63756359 +0000 UTC m=+1834.586211984"
Oct 31 01:51:16.654099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.651358    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"]
Oct 31 01:51:16.656824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.656795    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kvnjl"]
Oct 31 01:51:16.673347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.672606    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"]
Oct 31 01:51:16.675972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:16.675951    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-bl8fj"]
Oct 31 01:51:17.637721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:17.637155    5293 generic.go:356] "Generic (PLEG): container finished" podID="b73bc331-24f6-4135-bee9-787caba8cfe3" containerID="a8fc619eb86828af1858d3cc7842cbb6bfa063faf96f71d497602f872fe45377" exitCode=0
Oct 31 01:51:17.637721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:17.637278    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn" event={"ID":"b73bc331-24f6-4135-bee9-787caba8cfe3","Type":"ContainerDied","Data":"a8fc619eb86828af1858d3cc7842cbb6bfa063faf96f71d497602f872fe45377"}
Oct 31 01:51:17.643499 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:17.643121    5293 generic.go:356] "Generic (PLEG): container finished" podID="18c50da9-caab-4682-8b9b-50876f0a07ae" containerID="75f9a3ab7a0a045d3b00d0175315e1b2a730acc4e5b43f8a40fc0c16c38d5c59" exitCode=0
Oct 31 01:51:17.643499 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:17.643327    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t" event={"ID":"18c50da9-caab-4682-8b9b-50876f0a07ae","Type":"ContainerDied","Data":"75f9a3ab7a0a045d3b00d0175315e1b2a730acc4e5b43f8a40fc0c16c38d5c59"}
Oct 31 01:51:17.670312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:17.670231    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:51:17.670312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:17.670299    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:51:18.156893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.156852    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:51:18.178555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.178515    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" path="/var/lib/kubelet/pods/aff97f77-3447-4ff9-91ea-d7dda29aa919/volumes"
Oct 31 01:51:18.179616 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.179586    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d9c586-60ff-4ad2-9f84-f9292aa5b28e" path="/var/lib/kubelet/pods/b9d9c586-60ff-4ad2-9f84-f9292aa5b28e/volumes"
Oct 31 01:51:18.264466 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264349    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a96b284-a772-4d38-aac9-665181786781-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a96b284-a772-4d38-aac9-665181786781-istiod-ca-cert\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.264466 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264423    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-socket\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.264692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264473    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a96b284-a772-4d38-aac9-665181786781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a96b284-a772-4d38-aac9-665181786781-istio-podinfo\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.264692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264526    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-credential-socket\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.264692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264592    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-kube-api-access-fz662\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-kube-api-access-fz662\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.264853 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264754    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-istio-token\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-istio-token\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.264853 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264764    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a96b284-a772-4d38-aac9-665181786781-istiod-ca-cert" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:18.264960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264838    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-certs\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.264960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264946    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-data\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.265081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.264987    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-envoy\") pod \"5a96b284-a772-4d38-aac9-665181786781\" (UID: \"5a96b284-a772-4d38-aac9-665181786781\") "
Oct 31 01:51:18.265848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.265732    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-data" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:18.265848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.265778    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-credential-socket" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:18.266139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.265996    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.266139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.266018    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a96b284-a772-4d38-aac9-665181786781-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.266139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.266033    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.266311 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.266150    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-certs" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:18.266311 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.266203    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-socket" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:18.267510 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.267462    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-istio-token" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:18.267797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.267762    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a96b284-a772-4d38-aac9-665181786781-istio-podinfo" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:18.267904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.267883    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-kube-api-access-fz662" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "kube-api-access-fz662". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:18.268353 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.268330    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-envoy" pod "5a96b284-a772-4d38-aac9-665181786781" (UID: "5a96b284-a772-4d38-aac9-665181786781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:18.368132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.367686    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.368132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.367715    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.368132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.367725    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.368132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.367734    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a96b284-a772-4d38-aac9-665181786781-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.368132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.367745    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a96b284-a772-4d38-aac9-665181786781-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.368132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.367754    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz662\" (UniqueName: \"kubernetes.io/projected/5a96b284-a772-4d38-aac9-665181786781-kube-api-access-fz662\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:18.659379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.659338    5293 generic.go:356] "Generic (PLEG): container finished" podID="5a96b284-a772-4d38-aac9-665181786781" containerID="cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514" exitCode=0
Oct 31 01:51:18.659998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.659380    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" event={"ID":"5a96b284-a772-4d38-aac9-665181786781","Type":"ContainerDied","Data":"cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514"}
Oct 31 01:51:18.660135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.660026    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr" event={"ID":"5a96b284-a772-4d38-aac9-665181786781","Type":"ContainerDied","Data":"8530a12c28ad2356ce250ffd0f174e05db717c2fa126ae0a8c8f8d288948aaac"}
Oct 31 01:51:18.660135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.660073    5293 scope.go:122] "RemoveContainer" containerID="cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514"
Oct 31 01:51:18.660135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.659409    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"
Oct 31 01:51:18.680659 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.680623    5293 scope.go:122] "RemoveContainer" containerID="80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d"
Oct 31 01:51:18.699222 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.698844    5293 scope.go:122] "RemoveContainer" containerID="cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514"
Oct 31 01:51:18.699740 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:18.699470    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514\": container with ID starting with cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514 not found: ID does not exist" containerID="cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514"
Oct 31 01:51:18.699740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.699505    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514"} err="failed to get container status \"cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514\": rpc error: code = NotFound desc = could not find container \"cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514\": container with ID starting with cc55aae9b97921306b84fc6918974c0ead15eef375f5fb8c1eb61f5390792514 not found: ID does not exist"
Oct 31 01:51:18.699740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.699533    5293 scope.go:122] "RemoveContainer" containerID="80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d"
Oct 31 01:51:18.700229 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:18.700169    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d\": container with ID starting with 80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d not found: ID does not exist" containerID="80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d"
Oct 31 01:51:18.700229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.700199    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d"} err="failed to get container status \"80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d\": rpc error: code = NotFound desc = could not find container \"80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d\": container with ID starting with 80d9d6a62602fc8fa1153b509a1e3dbcb4b6c6f0821666ca6e479295f2eb6c3d not found: ID does not exist"
Oct 31 01:51:18.708061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.702636    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"]
Oct 31 01:51:18.711730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.711710    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-073e24-openshift-default-5d47c5496f-r6xpr"]
Oct 31 01:51:18.992241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:18.991952    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:19.033978 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.033883    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:19.081093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.081015    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-openshift-ca\") pod \"18c50da9-caab-4682-8b9b-50876f0a07ae\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") "
Oct 31 01:51:19.081286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.081142    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c50da9-caab-4682-8b9b-50876f0a07ae-kube-api-access-rzff6\" (UniqueName: \"kubernetes.io/projected/18c50da9-caab-4682-8b9b-50876f0a07ae-kube-api-access-rzff6\") pod \"18c50da9-caab-4682-8b9b-50876f0a07ae\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") "
Oct 31 01:51:19.081286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.081185    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-xks-ca\") pod \"18c50da9-caab-4682-8b9b-50876f0a07ae\" (UID: \"18c50da9-caab-4682-8b9b-50876f0a07ae\") "
Oct 31 01:51:19.081665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.081639    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-openshift-ca" pod "18c50da9-caab-4682-8b9b-50876f0a07ae" (UID: "18c50da9-caab-4682-8b9b-50876f0a07ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:19.082721 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.082691    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:19.084462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.084431    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c50da9-caab-4682-8b9b-50876f0a07ae-kube-api-access-rzff6" pod "18c50da9-caab-4682-8b9b-50876f0a07ae" (UID: "18c50da9-caab-4682-8b9b-50876f0a07ae"). InnerVolumeSpecName "kube-api-access-rzff6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:19.085441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.085410    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-xks-ca" pod "18c50da9-caab-4682-8b9b-50876f0a07ae" (UID: "18c50da9-caab-4682-8b9b-50876f0a07ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:19.189005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.184773    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-openshift-ca\") pod \"b73bc331-24f6-4135-bee9-787caba8cfe3\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") "
Oct 31 01:51:19.189005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.185584    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-xks-ca\") pod \"b73bc331-24f6-4135-bee9-787caba8cfe3\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") "
Oct 31 01:51:19.189005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.185744    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73bc331-24f6-4135-bee9-787caba8cfe3-kube-api-access-nt7wv\" (UniqueName: \"kubernetes.io/projected/b73bc331-24f6-4135-bee9-787caba8cfe3-kube-api-access-nt7wv\") pod \"b73bc331-24f6-4135-bee9-787caba8cfe3\" (UID: \"b73bc331-24f6-4135-bee9-787caba8cfe3\") "
Oct 31 01:51:19.189005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.186244    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-openshift-ca" pod "b73bc331-24f6-4135-bee9-787caba8cfe3" (UID: "b73bc331-24f6-4135-bee9-787caba8cfe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:19.189005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.186567    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzff6\" (UniqueName: \"kubernetes.io/projected/18c50da9-caab-4682-8b9b-50876f0a07ae-kube-api-access-rzff6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:19.189005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.186589    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18c50da9-caab-4682-8b9b-50876f0a07ae-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:19.189005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.186604    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:19.189516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.189465    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73bc331-24f6-4135-bee9-787caba8cfe3-kube-api-access-nt7wv" pod "b73bc331-24f6-4135-bee9-787caba8cfe3" (UID: "b73bc331-24f6-4135-bee9-787caba8cfe3"). InnerVolumeSpecName "kube-api-access-nt7wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:19.197845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.189998    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-xks-ca" pod "b73bc331-24f6-4135-bee9-787caba8cfe3" (UID: "b73bc331-24f6-4135-bee9-787caba8cfe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:19.287764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.287725    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73bc331-24f6-4135-bee9-787caba8cfe3-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:19.287764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.287762    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt7wv\" (UniqueName: \"kubernetes.io/projected/b73bc331-24f6-4135-bee9-787caba8cfe3-kube-api-access-nt7wv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:19.558670 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.552495    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"]
Oct 31 01:51:19.607758 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.579429    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e52ba1-4wj79"]
Oct 31 01:51:19.890421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.883225    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn" event={"ID":"b73bc331-24f6-4135-bee9-787caba8cfe3","Type":"ContainerDied","Data":"496c7ae6e5b7d1aadabed252d2abd83438ffe8c2bf21d126d0d23d35abc7e554"}
Oct 31 01:51:19.890421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.883316    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="496c7ae6e5b7d1aadabed252d2abd83438ffe8c2bf21d126d0d23d35abc7e554"
Oct 31 01:51:19.890421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.883417    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"
Oct 31 01:51:19.920279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.920233    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t" event={"ID":"18c50da9-caab-4682-8b9b-50876f0a07ae","Type":"ContainerDied","Data":"544ec8bad5c3caf208ce9d8086b04980b14a25abcedab4a51f76b9362784bd7b"}
Oct 31 01:51:19.920448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.920307    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="544ec8bad5c3caf208ce9d8086b04980b14a25abcedab4a51f76b9362784bd7b"
Oct 31 01:51:19.920448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:19.920406    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"
Oct 31 01:51:20.180114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.180029    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313e19da-61b5-41ba-b65b-ad985a697ed1" path="/var/lib/kubelet/pods/313e19da-61b5-41ba-b65b-ad985a697ed1/volumes"
Oct 31 01:51:20.180862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.180840    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a96b284-a772-4d38-aac9-665181786781" path="/var/lib/kubelet/pods/5a96b284-a772-4d38-aac9-665181786781/volumes"
Oct 31 01:51:20.938112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.937999    5293 generic.go:356] "Generic (PLEG): container finished" podID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerID="94555fa21e87680ce571eeb91cc4e3cb65bca72d71e2877b84f9ed98d71f9be5" exitCode=0
Oct 31 01:51:20.938112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.938074    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" event={"ID":"0a02efa4-83e1-4b47-a649-c3f3eaccb48f","Type":"ContainerDied","Data":"94555fa21e87680ce571eeb91cc4e3cb65bca72d71e2877b84f9ed98d71f9be5"}
Oct 31 01:51:20.938647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.938630    5293 scope.go:122] "RemoveContainer" containerID="94555fa21e87680ce571eeb91cc4e3cb65bca72d71e2877b84f9ed98d71f9be5"
Oct 31 01:51:20.940547 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.940515    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:20.940665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.940611    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:20.940665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.940656    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:20.940909 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.940888    5293 generic.go:356] "Generic (PLEG): container finished" podID="1b865fc7-6a25-467c-b154-45b71658520c" containerID="86c8a4e0628769a4bc09c955e834d9843ab7c45be5753ea4bf6b4c30c27d3ccf" exitCode=0
Oct 31 01:51:20.940968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.940949    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" event={"ID":"1b865fc7-6a25-467c-b154-45b71658520c","Type":"ContainerDied","Data":"86c8a4e0628769a4bc09c955e834d9843ab7c45be5753ea4bf6b4c30c27d3ccf"}
Oct 31 01:51:20.941425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.941399    5293 scope.go:122] "RemoveContainer" containerID="86c8a4e0628769a4bc09c955e834d9843ab7c45be5753ea4bf6b4c30c27d3ccf"
Oct 31 01:51:20.943108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.942984    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:20.943198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.943105    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:20.943198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:20.943160    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:21.978269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:21.978235    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" event={"ID":"0a02efa4-83e1-4b47-a649-c3f3eaccb48f","Type":"ContainerStarted","Data":"15620de740c472c9a37c089ffacc4e6b1cd6270c6638af152391640dfa37e9da"}
Oct 31 01:51:21.981684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:21.981655    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" event={"ID":"1b865fc7-6a25-467c-b154-45b71658520c","Type":"ContainerStarted","Data":"debe4a6c1264ca2755db3ac5233026de6ab150c20f3a750c7aa41da9ce76bf89"}
Oct 31 01:51:22.766640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:22.766607    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:51:22.766967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:22.766923    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:51:22.767218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:22.767198    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:51:22.972163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:22.972135    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:51:22.982209 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:22.975829    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 01:51:22.982209 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:22.975892    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 01:51:23.767554 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:23.767517    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:51:23.767737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:23.767570    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:51:23.973083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:23.972522    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 01:51:23.973083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:23.972579    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 01:51:24.766251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:24.766217    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:51:24.766858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:24.766825    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:51:24.972142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:24.972112    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 01:51:24.972318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:24.972159    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 01:51:25.767146 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:25.767109    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:51:25.767605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:25.767183    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:51:25.831313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:25.831278    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"]
Oct 31 01:51:25.831791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:25.831758    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy" containerID="cri-o://bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9" gracePeriod=30
Oct 31 01:51:25.877606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:25.877162    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"]
Oct 31 01:51:25.877606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:25.877423    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy" containerID="cri-o://2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132" gracePeriod=30
Oct 31 01:51:25.972457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:25.971992    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 01:51:25.972457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:25.972071    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 01:51:26.767078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:26.766429    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:51:26.767478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:26.767429    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:51:26.953063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:26.952094    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"]
Oct 31 01:51:26.974377 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:26.974343    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 01:51:26.974566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:26.974402    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 01:51:27.504185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.492319    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90c4d1-b6r2t"]
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.745711    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"]
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746721    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746742    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746769    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746777    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746827    5293 cpu_manager.go:414] "containerMap: removing container" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746835    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746847    5293 cpu_manager.go:414] "containerMap: removing container" podUID="18c50da9-caab-4682-8b9b-50876f0a07ae" containerName="revoke-keys"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746854    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c50da9-caab-4682-8b9b-50876f0a07ae" containerName="revoke-keys"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746873    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b73bc331-24f6-4135-bee9-787caba8cfe3" containerName="revoke-keys"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.746880    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73bc331-24f6-4135-bee9-787caba8cfe3" containerName="revoke-keys"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.747024    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff97f77-3447-4ff9-91ea-d7dda29aa919" containerName="istio-proxy"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.747055    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c50da9-caab-4682-8b9b-50876f0a07ae" containerName="revoke-keys"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.747068    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73bc331-24f6-4135-bee9-787caba8cfe3" containerName="revoke-keys"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.747085    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy"
Oct 31 01:51:27.750064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.747098    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a96b284-a772-4d38-aac9-665181786781" containerName="istio-proxy"
Oct 31 01:51:27.762128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.756360    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:27.766678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.762769    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:51:27.766678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.763569    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:51:27.766678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.763630    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"]
Oct 31 01:51:27.769617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.767560    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:51:27.769617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.767607    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:51:27.857066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.848506    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:51:27.857066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.848602    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:51:27.857066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.849590    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:27.857066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.849659    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ccl\" (UniqueName: \"kubernetes.io/projected/034d73b8-d39e-4a17-9370-b46a9a058934-kube-api-access-s2ccl\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:27.857066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:27.853092    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:28.031120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.013792    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:28.031120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.013879    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ccl\" (UniqueName: \"kubernetes.io/projected/034d73b8-d39e-4a17-9370-b46a9a058934-kube-api-access-s2ccl\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:28.031120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.014068    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:28.060066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.058149    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:28.097062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.096826    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 01:51:28.097062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.096883    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 01:51:28.119065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.109730    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:28.154064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.147009    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ccl\" (UniqueName: \"kubernetes.io/projected/034d73b8-d39e-4a17-9370-b46a9a058934-kube-api-access-s2ccl\") pod \"maas-api-revoke-sub-373189b1-sjdgn\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:28.272067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.265856    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c50da9-caab-4682-8b9b-50876f0a07ae" path="/var/lib/kubelet/pods/18c50da9-caab-4682-8b9b-50876f0a07ae/volumes"
Oct 31 01:51:28.433313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.433277    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:28.885132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:28.870779    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"]
Oct 31 01:51:29.297065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:29.296676    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:51:29.297990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:29.297967    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:51:29.300269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:29.300243    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:51:29.341112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:29.341082    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:51:29.343097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:29.342428    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:51:29.346777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:29.346749    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:51:29.564996 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:29.564955    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn" event={"ID":"034d73b8-d39e-4a17-9370-b46a9a058934","Type":"ContainerStarted","Data":"f0114818e011626232fb9866de7da52d9e8c3c05ec29f045dbceb362a491e6bf"}
Oct 31 01:51:30.589128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:30.587809    5293 generic.go:356] "Generic (PLEG): container finished" podID="034d73b8-d39e-4a17-9370-b46a9a058934" containerID="3df569da3626e2c81452ab5ad92a4ac258fa1f5ebe927b3fa9a898d24c3169a3" exitCode=0
Oct 31 01:51:30.589630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:30.589144    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn" event={"ID":"034d73b8-d39e-4a17-9370-b46a9a058934","Type":"ContainerDied","Data":"3df569da3626e2c81452ab5ad92a4ac258fa1f5ebe927b3fa9a898d24c3169a3"}
Oct 31 01:51:31.095169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.095128    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:51:31.223474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.223446    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:51:31.271302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271261    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-certs\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.271549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271367    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-podinfo\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.271549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271436    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-credential-socket\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.271549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271522    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istiod-ca-cert\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.271799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271667    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-kube-api-access-llxmc\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-kube-api-access-llxmc\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.271799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271664    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-certs" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.271799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271738    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-envoy\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.271799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271767    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-credential-socket" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.272013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.271784    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-token\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-token\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.272110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.272018    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istiod-ca-cert" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:31.272379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.272245    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-data\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.272502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.272484    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-socket\") pod \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\" (UID: \"3fe2189d-b5f0-4482-a55f-bd6056e83e52\") "
Oct 31 01:51:31.273187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.273140    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-socket" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.273364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.273330    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-data" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274314    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-podinfo" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274457    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274476    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274493    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274506    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274520    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274533    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274541    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-envoy" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.274937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.274721    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-token" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:31.275722 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.275700    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-kube-api-access-llxmc" pod "3fe2189d-b5f0-4482-a55f-bd6056e83e52" (UID: "3fe2189d-b5f0-4482-a55f-bd6056e83e52"). InnerVolumeSpecName "kube-api-access-llxmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:31.378438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378407    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-kube-api-access-wsr29\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-kube-api-access-wsr29\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.378585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378505    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ade77b0-64a0-447b-ba6a-a8199bb33298-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ade77b0-64a0-447b-ba6a-a8199bb33298-istiod-ca-cert\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.378585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378553    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-envoy\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.378676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378634    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-data\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.378714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378700    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-podinfo\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.378781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378766    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-certs\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.378925 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378908    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-credential-socket\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.378988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378944    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-token\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-token\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.378988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.378960    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ade77b0-64a0-447b-ba6a-a8199bb33298-istiod-ca-cert" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:31.379171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.379015    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-socket\") pod \"9ade77b0-64a0-447b-ba6a-a8199bb33298\" (UID: \"9ade77b0-64a0-447b-ba6a-a8199bb33298\") "
Oct 31 01:51:31.379293 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.379268    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-data" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.379293 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.379283    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-certs" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.379479 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.379461    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-socket" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.379664 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.379642    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-credential-socket" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.380204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.380033    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ade77b0-64a0-447b-ba6a-a8199bb33298-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.380204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.380092    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.380204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.380108    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.380204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.380123    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.380204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.380137    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.380204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.380153    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llxmc\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-kube-api-access-llxmc\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.380204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.380168    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.380204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.380182    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe2189d-b5f0-4482-a55f-bd6056e83e52-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.381333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.381309    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-envoy" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:31.381427 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.381393    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-kube-api-access-wsr29" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "kube-api-access-wsr29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:31.381594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.381574    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-podinfo" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:31.381730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.381712    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-token" pod "9ade77b0-64a0-447b-ba6a-a8199bb33298" (UID: "9ade77b0-64a0-447b-ba6a-a8199bb33298"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:31.481782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.481710    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.481782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.481739    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.481782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.481752    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.481782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.481761    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsr29\" (UniqueName: \"kubernetes.io/projected/9ade77b0-64a0-447b-ba6a-a8199bb33298-kube-api-access-wsr29\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:31.610065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.608246    5293 generic.go:356] "Generic (PLEG): container finished" podID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerID="bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9" exitCode=0
Oct 31 01:51:31.610065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.608344    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" event={"ID":"3fe2189d-b5f0-4482-a55f-bd6056e83e52","Type":"ContainerDied","Data":"bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9"}
Oct 31 01:51:31.610065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.608399    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p" event={"ID":"3fe2189d-b5f0-4482-a55f-bd6056e83e52","Type":"ContainerDied","Data":"f8c8310a68f133bb5bc172ac55b378fe0cbfe730a3f38cdaf3bc95368a5f4cbc"}
Oct 31 01:51:31.610065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.608425    5293 scope.go:122] "RemoveContainer" containerID="bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9"
Oct 31 01:51:31.610065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.608585    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"
Oct 31 01:51:31.630083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.625744    5293 generic.go:356] "Generic (PLEG): container finished" podID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerID="2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132" exitCode=0
Oct 31 01:51:31.630083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.626343    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"
Oct 31 01:51:31.630779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.630411    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" event={"ID":"9ade77b0-64a0-447b-ba6a-a8199bb33298","Type":"ContainerDied","Data":"2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132"}
Oct 31 01:51:31.630779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.630487    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd" event={"ID":"9ade77b0-64a0-447b-ba6a-a8199bb33298","Type":"ContainerDied","Data":"1db8729efd0aab2cd0b50cc6ca502de633131be42bebc563bc624c34416c59e5"}
Oct 31 01:51:31.751218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.750451    5293 scope.go:122] "RemoveContainer" containerID="bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9"
Oct 31 01:51:31.751218 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:31.750999    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9\": container with ID starting with bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9 not found: ID does not exist" containerID="bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9"
Oct 31 01:51:31.751218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.751054    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9"} err="failed to get container status \"bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9\": rpc error: code = NotFound desc = could not find container \"bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9\": container with ID starting with bf17b3e6f223302943bd6be7f72f497bd8430b0e10cf60806305b4505e6f73d9 not found: ID does not exist"
Oct 31 01:51:31.751218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.751079    5293 scope.go:122] "RemoveContainer" containerID="2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132"
Oct 31 01:51:31.792351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.792318    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"]
Oct 31 01:51:31.802386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.802113    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-28fqd"]
Oct 31 01:51:31.802386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.802223    5293 scope.go:122] "RemoveContainer" containerID="2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132"
Oct 31 01:51:31.802881 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:31.802766    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132\": container with ID starting with 2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132 not found: ID does not exist" containerID="2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132"
Oct 31 01:51:31.802881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.802808    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132"} err="failed to get container status \"2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132\": rpc error: code = NotFound desc = could not find container \"2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132\": container with ID starting with 2fc57958a61cb11a439bdb455fbce85ee4fe94bf998c306dae14b553e3c04132 not found: ID does not exist"
Oct 31 01:51:31.814586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.814554    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"]
Oct 31 01:51:31.816458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.816432    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-xrg6p"]
Oct 31 01:51:31.943138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:31.943110    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:32.120481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.120391    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034d73b8-d39e-4a17-9370-b46a9a058934-kube-api-access-s2ccl\" (UniqueName: \"kubernetes.io/projected/034d73b8-d39e-4a17-9370-b46a9a058934-kube-api-access-s2ccl\") pod \"034d73b8-d39e-4a17-9370-b46a9a058934\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") "
Oct 31 01:51:32.120654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.120587    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-openshift-ca\") pod \"034d73b8-d39e-4a17-9370-b46a9a058934\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") "
Oct 31 01:51:32.120730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.120697    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-xks-ca\") pod \"034d73b8-d39e-4a17-9370-b46a9a058934\" (UID: \"034d73b8-d39e-4a17-9370-b46a9a058934\") "
Oct 31 01:51:32.121014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.120983    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-openshift-ca" pod "034d73b8-d39e-4a17-9370-b46a9a058934" (UID: "034d73b8-d39e-4a17-9370-b46a9a058934"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:32.121408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.121386    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.122781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.122753    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034d73b8-d39e-4a17-9370-b46a9a058934-kube-api-access-s2ccl" pod "034d73b8-d39e-4a17-9370-b46a9a058934" (UID: "034d73b8-d39e-4a17-9370-b46a9a058934"). InnerVolumeSpecName "kube-api-access-s2ccl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:32.122877 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.122836    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-xks-ca" pod "034d73b8-d39e-4a17-9370-b46a9a058934" (UID: "034d73b8-d39e-4a17-9370-b46a9a058934"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:32.176624 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.176594    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" path="/var/lib/kubelet/pods/3fe2189d-b5f0-4482-a55f-bd6056e83e52/volumes"
Oct 31 01:51:32.177127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.177113    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" path="/var/lib/kubelet/pods/9ade77b0-64a0-447b-ba6a-a8199bb33298/volumes"
Oct 31 01:51:32.223225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.223031    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ccl\" (UniqueName: \"kubernetes.io/projected/034d73b8-d39e-4a17-9370-b46a9a058934-kube-api-access-s2ccl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.223225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.223107    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034d73b8-d39e-4a17-9370-b46a9a058934-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.740077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.740013    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"
Oct 31 01:51:32.740559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.740086    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn" event={"ID":"034d73b8-d39e-4a17-9370-b46a9a058934","Type":"ContainerDied","Data":"f0114818e011626232fb9866de7da52d9e8c3c05ec29f045dbceb362a491e6bf"}
Oct 31 01:51:32.740559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.740497    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0114818e011626232fb9866de7da52d9e8c3c05ec29f045dbceb362a491e6bf"
Oct 31 01:51:32.779355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.779323    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"]
Oct 31 01:51:32.779978 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.779961    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy"
Oct 31 01:51:32.779978 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.779979    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy"
Oct 31 01:51:32.780128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.780029    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy"
Oct 31 01:51:32.780128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.780049    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy"
Oct 31 01:51:32.780128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.780082    5293 cpu_manager.go:414] "containerMap: removing container" podUID="034d73b8-d39e-4a17-9370-b46a9a058934" containerName="revoke-keys"
Oct 31 01:51:32.780128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.780088    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034d73b8-d39e-4a17-9370-b46a9a058934" containerName="revoke-keys"
Oct 31 01:51:32.780249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.780185    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ade77b0-64a0-447b-ba6a-a8199bb33298" containerName="istio-proxy"
Oct 31 01:51:32.780249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.780196    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="034d73b8-d39e-4a17-9370-b46a9a058934" containerName="revoke-keys"
Oct 31 01:51:32.780249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.780204    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe2189d-b5f0-4482-a55f-bd6056e83e52" containerName="istio-proxy"
Oct 31 01:51:32.784922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.784897    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:32.789941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.789924    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:51:32.790444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.790406    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:51:32.791277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.791251    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:51:32.791391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.791353    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:51:32.794335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.794313    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"]
Oct 31 01:51:32.893344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.893309    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:32.893496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.893378    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4th\" (UniqueName: \"kubernetes.io/projected/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-kube-api-access-gt4th\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:32.893562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.893505    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:32.995186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.995105    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4th\" (UniqueName: \"kubernetes.io/projected/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-kube-api-access-gt4th\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:32.995370 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.995210    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:32.995448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.995427    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:32.995839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.995810    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:32.997921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:32.997899    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:33.006525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.006501    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4th\" (UniqueName: \"kubernetes.io/projected/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-kube-api-access-gt4th\") pod \"maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:33.101512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.101481    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:33.388904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.388876    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"]
Oct 31 01:51:33.390483 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:51:33.390453    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bdc788_cd3b_424e_bca6_52a80c14b7a3.slice/crio-4bf83cc3f1a75cb50cde92bf0bdf7279056ed444d7f41b7356218ab11c6d976c WatchSource:0}: Error finding container 4bf83cc3f1a75cb50cde92bf0bdf7279056ed444d7f41b7356218ab11c6d976c: Status 404 returned error can't find the container with id 4bf83cc3f1a75cb50cde92bf0bdf7279056ed444d7f41b7356218ab11c6d976c
Oct 31 01:51:33.657119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.657086    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"]
Oct 31 01:51:33.662582 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.662563    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:33.667717 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.667696    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a3ff0c\""
Oct 31 01:51:33.672156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.672135    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a3ff0c-dockercfg-c2djk\""
Oct 31 01:51:33.672718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.672692    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"]
Oct 31 01:51:33.695852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.695818    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"]
Oct 31 01:51:33.700058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.700023    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:33.724106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.711033    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"]
Oct 31 01:51:33.725506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.725476    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47tkb\" (UniqueName: \"kubernetes.io/projected/bc16f66e-3470-4ef7-8fec-046152e43eb6-kube-api-access-47tkb\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t\" (UID: \"bc16f66e-3470-4ef7-8fec-046152e43eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:33.725629 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.725545    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd\" (UID: \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:33.725685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.725634    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrvd\" (UniqueName: \"kubernetes.io/projected/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-kube-api-access-tbrvd\") pod \"payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd\" (UID: \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:33.725685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.725670    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16f66e-3470-4ef7-8fec-046152e43eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t\" (UID: \"bc16f66e-3470-4ef7-8fec-046152e43eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:33.914106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.906016    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrvd\" (UniqueName: \"kubernetes.io/projected/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-kube-api-access-tbrvd\") pod \"payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd\" (UID: \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:33.914106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.906091    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16f66e-3470-4ef7-8fec-046152e43eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t\" (UID: \"bc16f66e-3470-4ef7-8fec-046152e43eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:33.914106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.906265    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47tkb\" (UniqueName: \"kubernetes.io/projected/bc16f66e-3470-4ef7-8fec-046152e43eb6-kube-api-access-47tkb\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t\" (UID: \"bc16f66e-3470-4ef7-8fec-046152e43eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:33.914106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.906321    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd\" (UID: \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:33.914106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.907216    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd\" (UID: \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:33.914106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:33.908737    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16f66e-3470-4ef7-8fec-046152e43eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t\" (UID: \"bc16f66e-3470-4ef7-8fec-046152e43eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:34.000293 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:34.000241    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4" event={"ID":"d9bdc788-cd3b-424e-bca6-52a80c14b7a3","Type":"ContainerStarted","Data":"4bf83cc3f1a75cb50cde92bf0bdf7279056ed444d7f41b7356218ab11c6d976c"}
Oct 31 01:51:34.039062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:34.029693    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrvd\" (UniqueName: \"kubernetes.io/projected/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-kube-api-access-tbrvd\") pod \"payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd\" (UID: \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:34.039062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:34.030939    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47tkb\" (UniqueName: \"kubernetes.io/projected/bc16f66e-3470-4ef7-8fec-046152e43eb6-kube-api-access-47tkb\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t\" (UID: \"bc16f66e-3470-4ef7-8fec-046152e43eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:34.046543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:34.045018    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:34.279240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:34.279157    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:34.346238 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:51:34.345565    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc16f66e_3470_4ef7_8fec_046152e43eb6.slice/crio-fcd765e50e27e0b9eac37cac640b496693d954695557939db7f64fbcea35161c WatchSource:0}: Error finding container fcd765e50e27e0b9eac37cac640b496693d954695557939db7f64fbcea35161c: Status 404 returned error can't find the container with id fcd765e50e27e0b9eac37cac640b496693d954695557939db7f64fbcea35161c
Oct 31 01:51:34.377552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:34.356442    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"]
Oct 31 01:51:34.586117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:34.586084    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"]
Oct 31 01:51:34.588183 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:51:34.588156    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5681cd_75cf_44d6_8fc9_ab2347ab33bb.slice/crio-7a66d03aafe2adb5b8f11ca076f89bb1eeefb0e8c0b93da50cdef055792168f7 WatchSource:0}: Error finding container 7a66d03aafe2adb5b8f11ca076f89bb1eeefb0e8c0b93da50cdef055792168f7: Status 404 returned error can't find the container with id 7a66d03aafe2adb5b8f11ca076f89bb1eeefb0e8c0b93da50cdef055792168f7
Oct 31 01:51:35.084128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.078228    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd" event={"ID":"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb","Type":"ContainerStarted","Data":"387a89a3460babcd8785d9eaf8a365b96ae4b008900d6604a794f3fbe23f9a74"}
Oct 31 01:51:35.084128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.078298    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd" event={"ID":"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb","Type":"ContainerStarted","Data":"7a66d03aafe2adb5b8f11ca076f89bb1eeefb0e8c0b93da50cdef055792168f7"}
Oct 31 01:51:35.095248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.093590    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:35.108062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.106067    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t" event={"ID":"bc16f66e-3470-4ef7-8fec-046152e43eb6","Type":"ContainerStarted","Data":"1ec8d1884250aa8c5871e3c28fb66f31fa43b1c404067bbe97766df79eef45db"}
Oct 31 01:51:35.108062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.106148    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t" event={"ID":"bc16f66e-3470-4ef7-8fec-046152e43eb6","Type":"ContainerStarted","Data":"fcd765e50e27e0b9eac37cac640b496693d954695557939db7f64fbcea35161c"}
Oct 31 01:51:35.108062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.107004    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:35.115932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.115903    5293 generic.go:356] "Generic (PLEG): container finished" podID="d9bdc788-cd3b-424e-bca6-52a80c14b7a3" containerID="9547cf09b17016bf08b222c70cf779ebedd5e7f7e07ccd83e6d0781051cc2bab" exitCode=0
Oct 31 01:51:35.116234 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.116211    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4" event={"ID":"d9bdc788-cd3b-424e-bca6-52a80c14b7a3","Type":"ContainerDied","Data":"9547cf09b17016bf08b222c70cf779ebedd5e7f7e07ccd83e6d0781051cc2bab"}
Oct 31 01:51:35.128769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.128712    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd" podStartSLOduration=2.128692611 podStartE2EDuration="2.128692611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:35.114494159 +0000 UTC m=+1853.063142553" watchObservedRunningTime="2026-10-31 01:51:35.128692611 +0000 UTC m=+1853.077341005"
Oct 31 01:51:35.143946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:35.143809    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t" podStartSLOduration=2.143791158 podStartE2EDuration="2.143791158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:35.142389188 +0000 UTC m=+1853.091037585" watchObservedRunningTime="2026-10-31 01:51:35.143791158 +0000 UTC m=+1853.092439551"
Oct 31 01:51:36.383828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.383803    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:36.555521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.555491    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-openshift-ca\") pod \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") "
Oct 31 01:51:36.555730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.555665    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-xks-ca\") pod \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") "
Oct 31 01:51:36.555800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.555738    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-kube-api-access-gt4th\" (UniqueName: \"kubernetes.io/projected/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-kube-api-access-gt4th\") pod \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\" (UID: \"d9bdc788-cd3b-424e-bca6-52a80c14b7a3\") "
Oct 31 01:51:36.555859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.555839    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-openshift-ca" pod "d9bdc788-cd3b-424e-bca6-52a80c14b7a3" (UID: "d9bdc788-cd3b-424e-bca6-52a80c14b7a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:36.556370 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.556343    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:36.558103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.558070    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-xks-ca" pod "d9bdc788-cd3b-424e-bca6-52a80c14b7a3" (UID: "d9bdc788-cd3b-424e-bca6-52a80c14b7a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:36.558256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.558237    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-kube-api-access-gt4th" pod "d9bdc788-cd3b-424e-bca6-52a80c14b7a3" (UID: "d9bdc788-cd3b-424e-bca6-52a80c14b7a3"). InnerVolumeSpecName "kube-api-access-gt4th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:36.658548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.658510    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:36.658548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:36.658543    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt4th\" (UniqueName: \"kubernetes.io/projected/d9bdc788-cd3b-424e-bca6-52a80c14b7a3-kube-api-access-gt4th\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:37.001156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.001117    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"]
Oct 31 01:51:37.002131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.002108    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d9bdc788-cd3b-424e-bca6-52a80c14b7a3" containerName="revoke-keys"
Oct 31 01:51:37.002131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.002131    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bdc788-cd3b-424e-bca6-52a80c14b7a3" containerName="revoke-keys"
Oct 31 01:51:37.002334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.002321    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bdc788-cd3b-424e-bca6-52a80c14b7a3" containerName="revoke-keys"
Oct 31 01:51:37.006583 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.006559    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.010356 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.010325    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b8d890-openshift-default-dockercfg-fvs7g\""
Oct 31 01:51:37.033115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.029395    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"]
Oct 31 01:51:37.064438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.064406    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.064616 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.064573    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.064727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.064709    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.064786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.064769    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.064831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.064818    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8234a61-a840-4b87-b7d0-0868567a98e3-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.064871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.064842    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bj84\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-kube-api-access-9bj84\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.064973 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.064959    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.065087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.065029    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.065155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.065128    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169156    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169233    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169287    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169331    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169474    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169580    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169607    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169637    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8234a61-a840-4b87-b7d0-0868567a98e3-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.169667    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bj84\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-kube-api-access-9bj84\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.171073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.170466    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.179076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.171806    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.179076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.172164    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.179076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.172419    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.179076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.177698    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8234a61-a840-4b87-b7d0-0868567a98e3-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.219064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.212390    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.219064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.215191    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"
Oct 31 01:51:37.229409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.229362    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4" event={"ID":"d9bdc788-cd3b-424e-bca6-52a80c14b7a3","Type":"ContainerDied","Data":"4bf83cc3f1a75cb50cde92bf0bdf7279056ed444d7f41b7356218ab11c6d976c"}
Oct 31 01:51:37.229597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.229436    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bf83cc3f1a75cb50cde92bf0bdf7279056ed444d7f41b7356218ab11c6d976c"
Oct 31 01:51:37.283124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.275988    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bj84\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-kube-api-access-9bj84\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.283124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.279872    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.287070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.286654    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.331601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.330329    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:37.727229 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:51:37.727197    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8234a61_a840_4b87_b7d0_0868567a98e3.slice/crio-73ec13bb3e4a6bc37f11774ee869d3efc356cc93bf79796f68cce6309068443b WatchSource:0}: Error finding container 73ec13bb3e4a6bc37f11774ee869d3efc356cc93bf79796f68cce6309068443b: Status 404 returned error can't find the container with id 73ec13bb3e4a6bc37f11774ee869d3efc356cc93bf79796f68cce6309068443b
Oct 31 01:51:37.727714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.727653    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"]
Oct 31 01:51:37.730630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.730593    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:37.730755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.730677    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:37.730755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:37.730723    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:51:38.225305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:38.225268    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" event={"ID":"b8234a61-a840-4b87-b7d0-0868567a98e3","Type":"ContainerStarted","Data":"2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480"}
Oct 31 01:51:38.225514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:38.225337    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" event={"ID":"b8234a61-a840-4b87-b7d0-0868567a98e3","Type":"ContainerStarted","Data":"73ec13bb3e4a6bc37f11774ee869d3efc356cc93bf79796f68cce6309068443b"}
Oct 31 01:51:38.250601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:38.250555    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podStartSLOduration=2.250541069 podStartE2EDuration="2.250541069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:38.248488666 +0000 UTC m=+1856.197137057" watchObservedRunningTime="2026-10-31 01:51:38.250541069 +0000 UTC m=+1856.199189501"
Oct 31 01:51:38.331410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:38.331382    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:38.337492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:38.337465    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:39.234279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:39.234244    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:39.235337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:39.235317    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:51:41.779971 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:41.779936    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:51:42.855435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:42.855399    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"]
Oct 31 01:51:42.868001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:42.867976    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"]
Oct 31 01:51:44.220447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:44.220418    5293 scope.go:122] "RemoveContainer" containerID="4cd0361c8bbb98a4907c6ea464ab76e25bb56db44ef0407b16f4048465ffd46e"
Oct 31 01:51:44.231352 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:44.231329    5293 scope.go:122] "RemoveContainer" containerID="7fbb9f24b838b9271bbb10f567afc4908b0cf7483c163ac96d44ec0c073040c8"
Oct 31 01:51:44.240711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:44.240691    5293 scope.go:122] "RemoveContainer" containerID="4a8cd4b1bf37996ac583186eebb2126bb5fafb1ef03b5331f0d9a3ab121c1e34"
Oct 31 01:51:44.250504 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:44.250488    5293 scope.go:122] "RemoveContainer" containerID="487bc630ec07d1abdc88d9c0c4a466832aaa4aa07d1f059a05db18fb87740b8e"
Oct 31 01:51:44.259729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:44.259712    5293 scope.go:122] "RemoveContainer" containerID="e38c97cbd0894e18b19176d5d358108e0ddbe1506b09c183862dcb015ac6b3b3"
Oct 31 01:51:46.073064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:46.067100    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"]
Oct 31 01:51:46.128066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:46.122291    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ad416-hmdrd"]
Oct 31 01:51:46.241391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:46.241356    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabe9d99-a090-4e04-9f0d-2d064670c347" path="/var/lib/kubelet/pods/fabe9d99-a090-4e04-9f0d-2d064670c347/volumes"
Oct 31 01:51:46.241739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:46.241724    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"]
Oct 31 01:51:46.241806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:46.241757    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-611693-5wzz4"]
Oct 31 01:51:47.304344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:47.304314    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:47.318387 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:47.318361    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:47.419468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:47.419431    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"]
Oct 31 01:51:47.425856 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:47.425825    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-03004a-lsjvn"]
Oct 31 01:51:47.856004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:47.855968    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn" podUID="c911e715-4526-4f9a-9a31-68a9b2f38cee" containerName="payload-processing" containerID="cri-o://2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074" gracePeriod=25
Oct 31 01:51:47.868525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:47.868477    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm" podUID="5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8" containerName="payload-pre-processing" containerID="cri-o://a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3" gracePeriod=25
Oct 31 01:51:48.163219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.163184    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:51:48.175125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.175097    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73bc331-24f6-4135-bee9-787caba8cfe3" path="/var/lib/kubelet/pods/b73bc331-24f6-4135-bee9-787caba8cfe3/volumes"
Oct 31 01:51:48.175660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.175629    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bdc788-cd3b-424e-bca6-52a80c14b7a3" path="/var/lib/kubelet/pods/d9bdc788-cd3b-424e-bca6-52a80c14b7a3/volumes"
Oct 31 01:51:48.200514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.200490    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:51:48.233269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.233244    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c911e715-4526-4f9a-9a31-68a9b2f38cee-kube-api-access-zrxrx\" (UniqueName: \"kubernetes.io/projected/c911e715-4526-4f9a-9a31-68a9b2f38cee-kube-api-access-zrxrx\") pod \"c911e715-4526-4f9a-9a31-68a9b2f38cee\" (UID: \"c911e715-4526-4f9a-9a31-68a9b2f38cee\") "
Oct 31 01:51:48.233450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.233313    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c911e715-4526-4f9a-9a31-68a9b2f38cee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c911e715-4526-4f9a-9a31-68a9b2f38cee-plugins-config-volume\") pod \"c911e715-4526-4f9a-9a31-68a9b2f38cee\" (UID: \"c911e715-4526-4f9a-9a31-68a9b2f38cee\") "
Oct 31 01:51:48.233743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.233713    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c911e715-4526-4f9a-9a31-68a9b2f38cee-plugins-config-volume" pod "c911e715-4526-4f9a-9a31-68a9b2f38cee" (UID: "c911e715-4526-4f9a-9a31-68a9b2f38cee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:48.234142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.234065    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c911e715-4526-4f9a-9a31-68a9b2f38cee-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.235944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.235913    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c911e715-4526-4f9a-9a31-68a9b2f38cee-kube-api-access-zrxrx" pod "c911e715-4526-4f9a-9a31-68a9b2f38cee" (UID: "c911e715-4526-4f9a-9a31-68a9b2f38cee"). InnerVolumeSpecName "kube-api-access-zrxrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:48.335187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.335152    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-kube-api-access-n4lmt\" (UniqueName: \"kubernetes.io/projected/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-kube-api-access-n4lmt\") pod \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\" (UID: \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\") "
Oct 31 01:51:48.335581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.335216    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-plugins-config-volume\") pod \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\" (UID: \"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8\") "
Oct 31 01:51:48.335649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.335626    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-plugins-config-volume" pod "5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8" (UID: "5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:48.335723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.335707    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxrx\" (UniqueName: \"kubernetes.io/projected/c911e715-4526-4f9a-9a31-68a9b2f38cee-kube-api-access-zrxrx\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.337674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.337645    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-kube-api-access-n4lmt" pod "5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8" (UID: "5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8"). InnerVolumeSpecName "kube-api-access-n4lmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:48.417331 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.417292    5293 generic.go:356] "Generic (PLEG): container finished" podID="5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8" containerID="a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3" exitCode=0
Oct 31 01:51:48.417540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.417350    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm" event={"ID":"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8","Type":"ContainerDied","Data":"a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3"}
Oct 31 01:51:48.417540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.417354    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"
Oct 31 01:51:48.417540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.417434    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm" event={"ID":"5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8","Type":"ContainerDied","Data":"77e095db4d94529f8e06fb330ba33db95637af706fc13756e62d69bb558f5769"}
Oct 31 01:51:48.417540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.417459    5293 scope.go:122] "RemoveContainer" containerID="a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3"
Oct 31 01:51:48.419352 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.419324    5293 generic.go:356] "Generic (PLEG): container finished" podID="c911e715-4526-4f9a-9a31-68a9b2f38cee" containerID="2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074" exitCode=0
Oct 31 01:51:48.419490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.419381    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"
Oct 31 01:51:48.419490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.419402    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn" event={"ID":"c911e715-4526-4f9a-9a31-68a9b2f38cee","Type":"ContainerDied","Data":"2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074"}
Oct 31 01:51:48.419490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.419446    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn" event={"ID":"c911e715-4526-4f9a-9a31-68a9b2f38cee","Type":"ContainerDied","Data":"ba642ab8fe7123bc4aae7503b51bf251429643cd4b50705b18bba717fb61e6e8"}
Oct 31 01:51:48.429831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.429810    5293 scope.go:122] "RemoveContainer" containerID="a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3"
Oct 31 01:51:48.430185 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:48.430161    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3\": container with ID starting with a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3 not found: ID does not exist" containerID="a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3"
Oct 31 01:51:48.430261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.430194    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3"} err="failed to get container status \"a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3\": rpc error: code = NotFound desc = could not find container \"a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3\": container with ID starting with a0fd20d226e2c9c43d0a328e52c90058ccfd44c96c21320b935fce256ecb89b3 not found: ID does not exist"
Oct 31 01:51:48.430261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.430212    5293 scope.go:122] "RemoveContainer" containerID="2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074"
Oct 31 01:51:48.437774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.437750    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4lmt\" (UniqueName: \"kubernetes.io/projected/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-kube-api-access-n4lmt\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.437862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.437781    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.442649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.442230    5293 scope.go:122] "RemoveContainer" containerID="2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074"
Oct 31 01:51:48.484064 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:51:48.483190    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074\": container with ID starting with 2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074 not found: ID does not exist" containerID="2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074"
Oct 31 01:51:48.484064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.483246    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074"} err="failed to get container status \"2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074\": rpc error: code = NotFound desc = could not find container \"2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074\": container with ID starting with 2153d3e79fd66e68b3ef22c7f6295359c9e5bb7b906ade3be1e7590fb9289074 not found: ID does not exist"
Oct 31 01:51:48.510170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.510123    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"]
Oct 31 01:51:48.516296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.516102    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-611693-659f7dbbc-nv5rn"]
Oct 31 01:51:48.525206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.525167    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"]
Oct 31 01:51:48.530419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:48.530280    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-611693-7fcd9546c-sz9nm"]
Oct 31 01:51:50.175608 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:50.175579    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8" path="/var/lib/kubelet/pods/5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8/volumes"
Oct 31 01:51:50.176064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:50.176026    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c911e715-4526-4f9a-9a31-68a9b2f38cee" path="/var/lib/kubelet/pods/c911e715-4526-4f9a-9a31-68a9b2f38cee/volumes"
Oct 31 01:51:52.478495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:52.478467    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"]
Oct 31 01:51:52.492986 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:52.492960    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"]
Oct 31 01:51:53.015799 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.015496    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"]
Oct 31 01:51:53.307576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.294270    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"]
Oct 31 01:51:53.307576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.294538    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy" containerID="cri-o://15620de740c472c9a37c089ffacc4e6b1cd6270c6638af152391640dfa37e9da" gracePeriod=30
Oct 31 01:51:53.845677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.845641    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"]
Oct 31 01:51:53.846326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.846311    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c911e715-4526-4f9a-9a31-68a9b2f38cee" containerName="payload-processing"
Oct 31 01:51:53.846389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.846328    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c911e715-4526-4f9a-9a31-68a9b2f38cee" containerName="payload-processing"
Oct 31 01:51:53.846389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.846365    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8" containerName="payload-pre-processing"
Oct 31 01:51:53.846389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.846370    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8" containerName="payload-pre-processing"
Oct 31 01:51:53.846497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.846478    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c911e715-4526-4f9a-9a31-68a9b2f38cee" containerName="payload-processing"
Oct 31 01:51:53.846497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.846491    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5e2ab6-7c7c-48cb-8492-5cadbbe226e8" containerName="payload-pre-processing"
Oct 31 01:51:53.851327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.851309    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:53.855940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.855809    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:51:53.855940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.855909    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:51:53.860353 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.860180    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"]
Oct 31 01:51:53.894278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.894248    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:51:53.894443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.894315    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:51:53.982497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.982453    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:53.982692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.982512    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:53.982692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:53.982595    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5gx\" (UniqueName: \"kubernetes.io/projected/1f1511b2-c2be-457d-ac6d-91963d36f8f2-kube-api-access-9n5gx\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:54.089072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.088322    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:54.092061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.089699    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:54.092061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.089785    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5gx\" (UniqueName: \"kubernetes.io/projected/1f1511b2-c2be-457d-ac6d-91963d36f8f2-kube-api-access-9n5gx\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:54.094092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.093432    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:54.097140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.096264    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:54.108557 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.108535    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5gx\" (UniqueName: \"kubernetes.io/projected/1f1511b2-c2be-457d-ac6d-91963d36f8f2-kube-api-access-9n5gx\") pod \"maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:54.167643 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.167617    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:54.474154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.474122    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"]
Oct 31 01:51:54.475875 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:51:54.475845    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1511b2_c2be_457d_ac6d_91963d36f8f2.slice/crio-441db2f81967cc2859c5f826bdf1f083f1a3a6605cbabaa6c44efc1ac1199a29 WatchSource:0}: Error finding container 441db2f81967cc2859c5f826bdf1f083f1a3a6605cbabaa6c44efc1ac1199a29: Status 404 returned error can't find the container with id 441db2f81967cc2859c5f826bdf1f083f1a3a6605cbabaa6c44efc1ac1199a29
Oct 31 01:51:54.605128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:54.605091    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl" event={"ID":"1f1511b2-c2be-457d-ac6d-91963d36f8f2","Type":"ContainerStarted","Data":"441db2f81967cc2859c5f826bdf1f083f1a3a6605cbabaa6c44efc1ac1199a29"}
Oct 31 01:51:55.646462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:55.646426    5293 generic.go:356] "Generic (PLEG): container finished" podID="1f1511b2-c2be-457d-ac6d-91963d36f8f2" containerID="0b04f3f5a4885ac6da01b00da7ac52ad8787798a0614148c6578c136fb2ed792" exitCode=0
Oct 31 01:51:55.646850 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:55.646685    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl" event={"ID":"1f1511b2-c2be-457d-ac6d-91963d36f8f2","Type":"ContainerDied","Data":"0b04f3f5a4885ac6da01b00da7ac52ad8787798a0614148c6578c136fb2ed792"}
Oct 31 01:51:56.819158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.819127    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:56.865557 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.865520    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-openshift-ca\") pod \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") "
Oct 31 01:51:56.865726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.865570    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-xks-ca\") pod \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") "
Oct 31 01:51:56.865726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.865611    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1511b2-c2be-457d-ac6d-91963d36f8f2-kube-api-access-9n5gx\" (UniqueName: \"kubernetes.io/projected/1f1511b2-c2be-457d-ac6d-91963d36f8f2-kube-api-access-9n5gx\") pod \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\" (UID: \"1f1511b2-c2be-457d-ac6d-91963d36f8f2\") "
Oct 31 01:51:56.865954 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.865923    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-openshift-ca" pod "1f1511b2-c2be-457d-ac6d-91963d36f8f2" (UID: "1f1511b2-c2be-457d-ac6d-91963d36f8f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:56.866360 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.866331    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:56.867988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.867956    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1511b2-c2be-457d-ac6d-91963d36f8f2-kube-api-access-9n5gx" pod "1f1511b2-c2be-457d-ac6d-91963d36f8f2" (UID: "1f1511b2-c2be-457d-ac6d-91963d36f8f2"). InnerVolumeSpecName "kube-api-access-9n5gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:56.868378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.868351    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-xks-ca" pod "1f1511b2-c2be-457d-ac6d-91963d36f8f2" (UID: "1f1511b2-c2be-457d-ac6d-91963d36f8f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:56.968250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.968167    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1511b2-c2be-457d-ac6d-91963d36f8f2-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:56.968250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:56.968198    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n5gx\" (UniqueName: \"kubernetes.io/projected/1f1511b2-c2be-457d-ac6d-91963d36f8f2-kube-api-access-9n5gx\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:57.479106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.479069    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd" podUID="7d5681cd-75cf-44d6-8fc9-ab2347ab33bb" containerName="payload-processing" containerID="cri-o://387a89a3460babcd8785d9eaf8a365b96ae4b008900d6604a794f3fbe23f9a74" gracePeriod=25
Oct 31 01:51:57.493723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.493674    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t" podUID="bc16f66e-3470-4ef7-8fec-046152e43eb6" containerName="payload-pre-processing" containerID="cri-o://1ec8d1884250aa8c5871e3c28fb66f31fa43b1c404067bbe97766df79eef45db" gracePeriod=25
Oct 31 01:51:57.666326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.666293    5293 generic.go:356] "Generic (PLEG): container finished" podID="7d5681cd-75cf-44d6-8fc9-ab2347ab33bb" containerID="387a89a3460babcd8785d9eaf8a365b96ae4b008900d6604a794f3fbe23f9a74" exitCode=0
Oct 31 01:51:57.666498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.666366    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd" event={"ID":"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb","Type":"ContainerDied","Data":"387a89a3460babcd8785d9eaf8a365b96ae4b008900d6604a794f3fbe23f9a74"}
Oct 31 01:51:57.668154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.668133    5293 generic.go:356] "Generic (PLEG): container finished" podID="bc16f66e-3470-4ef7-8fec-046152e43eb6" containerID="1ec8d1884250aa8c5871e3c28fb66f31fa43b1c404067bbe97766df79eef45db" exitCode=0
Oct 31 01:51:57.668277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.668206    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t" event={"ID":"bc16f66e-3470-4ef7-8fec-046152e43eb6","Type":"ContainerDied","Data":"1ec8d1884250aa8c5871e3c28fb66f31fa43b1c404067bbe97766df79eef45db"}
Oct 31 01:51:57.669653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.669629    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl" event={"ID":"1f1511b2-c2be-457d-ac6d-91963d36f8f2","Type":"ContainerDied","Data":"441db2f81967cc2859c5f826bdf1f083f1a3a6605cbabaa6c44efc1ac1199a29"}
Oct 31 01:51:57.669782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.669666    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="441db2f81967cc2859c5f826bdf1f083f1a3a6605cbabaa6c44efc1ac1199a29"
Oct 31 01:51:57.669782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.669719    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"
Oct 31 01:51:57.772313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.772282    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:57.921538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.918211    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc16f66e-3470-4ef7-8fec-046152e43eb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16f66e-3470-4ef7-8fec-046152e43eb6-plugins-config-volume\") pod \"bc16f66e-3470-4ef7-8fec-046152e43eb6\" (UID: \"bc16f66e-3470-4ef7-8fec-046152e43eb6\") "
Oct 31 01:51:57.921538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.920617    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc16f66e-3470-4ef7-8fec-046152e43eb6-kube-api-access-47tkb\" (UniqueName: \"kubernetes.io/projected/bc16f66e-3470-4ef7-8fec-046152e43eb6-kube-api-access-47tkb\") pod \"bc16f66e-3470-4ef7-8fec-046152e43eb6\" (UID: \"bc16f66e-3470-4ef7-8fec-046152e43eb6\") "
Oct 31 01:51:57.930084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.929839    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc16f66e-3470-4ef7-8fec-046152e43eb6-kube-api-access-47tkb" pod "bc16f66e-3470-4ef7-8fec-046152e43eb6" (UID: "bc16f66e-3470-4ef7-8fec-046152e43eb6"). InnerVolumeSpecName "kube-api-access-47tkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:57.930493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.930471    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc16f66e-3470-4ef7-8fec-046152e43eb6-plugins-config-volume" pod "bc16f66e-3470-4ef7-8fec-046152e43eb6" (UID: "bc16f66e-3470-4ef7-8fec-046152e43eb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:57.977576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:57.977550    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:58.023468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.023371    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47tkb\" (UniqueName: \"kubernetes.io/projected/bc16f66e-3470-4ef7-8fec-046152e43eb6-kube-api-access-47tkb\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:58.023468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.023405    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc16f66e-3470-4ef7-8fec-046152e43eb6-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:58.124193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.124160    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-kube-api-access-tbrvd\" (UniqueName: \"kubernetes.io/projected/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-kube-api-access-tbrvd\") pod \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\" (UID: \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\") "
Oct 31 01:51:58.124444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.124428    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-plugins-config-volume\") pod \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\" (UID: \"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb\") "
Oct 31 01:51:58.124920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.124848    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-plugins-config-volume" pod "7d5681cd-75cf-44d6-8fc9-ab2347ab33bb" (UID: "7d5681cd-75cf-44d6-8fc9-ab2347ab33bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:58.125180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.125127    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:58.126673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.126641    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-kube-api-access-tbrvd" pod "7d5681cd-75cf-44d6-8fc9-ab2347ab33bb" (UID: "7d5681cd-75cf-44d6-8fc9-ab2347ab33bb"). InnerVolumeSpecName "kube-api-access-tbrvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:58.226374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.226341    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbrvd\" (UniqueName: \"kubernetes.io/projected/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb-kube-api-access-tbrvd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:58.295387 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.295311    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:51:58.693066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.689618    5293 generic.go:356] "Generic (PLEG): container finished" podID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerID="15620de740c472c9a37c089ffacc4e6b1cd6270c6638af152391640dfa37e9da" exitCode=0
Oct 31 01:51:58.693066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.689742    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" event={"ID":"0a02efa4-83e1-4b47-a649-c3f3eaccb48f","Type":"ContainerDied","Data":"15620de740c472c9a37c089ffacc4e6b1cd6270c6638af152391640dfa37e9da"}
Oct 31 01:51:58.693066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.689799    5293 scope.go:122] "RemoveContainer" containerID="94555fa21e87680ce571eeb91cc4e3cb65bca72d71e2877b84f9ed98d71f9be5"
Oct 31 01:51:58.697106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.695562    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd" event={"ID":"7d5681cd-75cf-44d6-8fc9-ab2347ab33bb","Type":"ContainerDied","Data":"7a66d03aafe2adb5b8f11ca076f89bb1eeefb0e8c0b93da50cdef055792168f7"}
Oct 31 01:51:58.697106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.695715    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"
Oct 31 01:51:58.708065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.698301    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t" event={"ID":"bc16f66e-3470-4ef7-8fec-046152e43eb6","Type":"ContainerDied","Data":"fcd765e50e27e0b9eac37cac640b496693d954695557939db7f64fbcea35161c"}
Oct 31 01:51:58.708065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.698465    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"
Oct 31 01:51:58.749210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.743810    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"]
Oct 31 01:51:58.749486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.749460    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-68cf79bdf5-6rlkd"]
Oct 31 01:51:58.752271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.752249    5293 scope.go:122] "RemoveContainer" containerID="387a89a3460babcd8785d9eaf8a365b96ae4b008900d6604a794f3fbe23f9a74"
Oct 31 01:51:58.758429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.758405    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"]
Oct 31 01:51:58.762665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.762639    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-d9ffcbff5-76q9t"]
Oct 31 01:51:58.763445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:58.763311    5293 scope.go:122] "RemoveContainer" containerID="1ec8d1884250aa8c5871e3c28fb66f31fa43b1c404067bbe97766df79eef45db"
Oct 31 01:51:59.054345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.054300    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:51:59.151067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.150333    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-socket\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.151067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.150460    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-certs\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.151067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.150500    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-credential-socket\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.151067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.150554    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-podinfo\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.151067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.150927    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-socket" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:59.151487 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.151197    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-credential-socket" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:59.151487 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.151476    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-certs" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:59.151613 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.151528    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-token\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-token\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.152244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.151831    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istiod-ca-cert\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.152244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.151940    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-kube-api-access-ngtqg\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-kube-api-access-ngtqg\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.152244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.151981    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-envoy\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.152244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.152033    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-data\") pod \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\" (UID: \"0a02efa4-83e1-4b47-a649-c3f3eaccb48f\") "
Oct 31 01:51:59.154361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.154057    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istiod-ca-cert" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:59.154361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.154089    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-data" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:59.154361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.154299    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.154361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.154318    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.154361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.154326    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.154361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.154335    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.154361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.154343    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.158259 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.157876    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-token" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:59.168843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.161275    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-podinfo" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:59.168843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.162578    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-envoy" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:59.168843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.164474    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-kube-api-access-ngtqg" pod "0a02efa4-83e1-4b47-a649-c3f3eaccb48f" (UID: "0a02efa4-83e1-4b47-a649-c3f3eaccb48f"). InnerVolumeSpecName "kube-api-access-ngtqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:59.301115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.287704    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.301115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.287738    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.301115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.287751    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngtqg\" (UniqueName: \"kubernetes.io/projected/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-kube-api-access-ngtqg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.301115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.287765    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a02efa4-83e1-4b47-a649-c3f3eaccb48f-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:59.751099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.749805    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j" event={"ID":"0a02efa4-83e1-4b47-a649-c3f3eaccb48f","Type":"ContainerDied","Data":"91c5ba048bd514fc1280d663e621209aa3c571c8b9769fa28b54721f8d5888da"}
Oct 31 01:51:59.751099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.749906    5293 scope.go:122] "RemoveContainer" containerID="15620de740c472c9a37c089ffacc4e6b1cd6270c6638af152391640dfa37e9da"
Oct 31 01:51:59.751099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.750081    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"
Oct 31 01:51:59.873185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.868306    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"]
Oct 31 01:51:59.882121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:51:59.878511    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-611693-openshift-default-7c79c8b6c7-k568j"]
Oct 31 01:52:00.174123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:00.174087    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" path="/var/lib/kubelet/pods/0a02efa4-83e1-4b47-a649-c3f3eaccb48f/volumes"
Oct 31 01:52:00.174742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:00.174720    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5681cd-75cf-44d6-8fc9-ab2347ab33bb" path="/var/lib/kubelet/pods/7d5681cd-75cf-44d6-8fc9-ab2347ab33bb/volumes"
Oct 31 01:52:00.175214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:00.175194    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc16f66e-3470-4ef7-8fec-046152e43eb6" path="/var/lib/kubelet/pods/bc16f66e-3470-4ef7-8fec-046152e43eb6/volumes"
Oct 31 01:52:01.982667 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.982636    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"]
Oct 31 01:52:01.983257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983242    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy"
Oct 31 01:52:01.983299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983258    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy"
Oct 31 01:52:01.983299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983289    5293 cpu_manager.go:414] "containerMap: removing container" podUID="bc16f66e-3470-4ef7-8fec-046152e43eb6" containerName="payload-pre-processing"
Oct 31 01:52:01.983299 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983295    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc16f66e-3470-4ef7-8fec-046152e43eb6" containerName="payload-pre-processing"
Oct 31 01:52:01.983396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983303    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1f1511b2-c2be-457d-ac6d-91963d36f8f2" containerName="revoke-keys"
Oct 31 01:52:01.983396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983308    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1511b2-c2be-457d-ac6d-91963d36f8f2" containerName="revoke-keys"
Oct 31 01:52:01.983396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983320    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7d5681cd-75cf-44d6-8fc9-ab2347ab33bb" containerName="payload-processing"
Oct 31 01:52:01.983396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983324    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5681cd-75cf-44d6-8fc9-ab2347ab33bb" containerName="payload-processing"
Oct 31 01:52:01.983516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983431    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy"
Oct 31 01:52:01.983516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983447    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc16f66e-3470-4ef7-8fec-046152e43eb6" containerName="payload-pre-processing"
Oct 31 01:52:01.983516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983454    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1511b2-c2be-457d-ac6d-91963d36f8f2" containerName="revoke-keys"
Oct 31 01:52:01.983516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.983461    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5681cd-75cf-44d6-8fc9-ab2347ab33bb" containerName="payload-processing"
Oct 31 01:52:01.988478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.988460    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:01.993241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.993216    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a3ff0c-dockercfg-79ljd\""
Oct 31 01:52:01.993241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.993215    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a3ff0c\""
Oct 31 01:52:01.998166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:01.997674    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"]
Oct 31 01:52:02.030901 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.030861    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"]
Oct 31 01:52:02.031828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.031803    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy"
Oct 31 01:52:02.031941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.031831    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy"
Oct 31 01:52:02.032101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.032083    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a02efa4-83e1-4b47-a649-c3f3eaccb48f" containerName="istio-proxy"
Oct 31 01:52:02.036434 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.036414    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:02.055362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.055336    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"]
Oct 31 01:52:02.064088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.057894    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfl8\" (UniqueName: \"kubernetes.io/projected/8351577d-def3-46f5-9ccd-30772fa5a5c9-kube-api-access-dkfl8\") pod \"payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs\" (UID: \"8351577d-def3-46f5-9ccd-30772fa5a5c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:02.064088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.058000    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bab320c-4018-44c6-a7cf-06f76f5f315a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr\" (UID: \"2bab320c-4018-44c6-a7cf-06f76f5f315a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:02.064088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.058185    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfxg9\" (UniqueName: \"kubernetes.io/projected/2bab320c-4018-44c6-a7cf-06f76f5f315a-kube-api-access-zfxg9\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr\" (UID: \"2bab320c-4018-44c6-a7cf-06f76f5f315a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:02.064088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.058276    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8351577d-def3-46f5-9ccd-30772fa5a5c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs\" (UID: \"8351577d-def3-46f5-9ccd-30772fa5a5c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:02.163065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.161608    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfl8\" (UniqueName: \"kubernetes.io/projected/8351577d-def3-46f5-9ccd-30772fa5a5c9-kube-api-access-dkfl8\") pod \"payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs\" (UID: \"8351577d-def3-46f5-9ccd-30772fa5a5c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:02.163065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.161701    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bab320c-4018-44c6-a7cf-06f76f5f315a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr\" (UID: \"2bab320c-4018-44c6-a7cf-06f76f5f315a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:02.163065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.161731    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfxg9\" (UniqueName: \"kubernetes.io/projected/2bab320c-4018-44c6-a7cf-06f76f5f315a-kube-api-access-zfxg9\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr\" (UID: \"2bab320c-4018-44c6-a7cf-06f76f5f315a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:02.163065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.161795    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8351577d-def3-46f5-9ccd-30772fa5a5c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs\" (UID: \"8351577d-def3-46f5-9ccd-30772fa5a5c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:02.163065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.162673    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8351577d-def3-46f5-9ccd-30772fa5a5c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs\" (UID: \"8351577d-def3-46f5-9ccd-30772fa5a5c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:02.176382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.164846    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bab320c-4018-44c6-a7cf-06f76f5f315a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr\" (UID: \"2bab320c-4018-44c6-a7cf-06f76f5f315a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:02.347940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.347855    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfl8\" (UniqueName: \"kubernetes.io/projected/8351577d-def3-46f5-9ccd-30772fa5a5c9-kube-api-access-dkfl8\") pod \"payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs\" (UID: \"8351577d-def3-46f5-9ccd-30772fa5a5c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:02.354107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.353994    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfxg9\" (UniqueName: \"kubernetes.io/projected/2bab320c-4018-44c6-a7cf-06f76f5f315a-kube-api-access-zfxg9\") pod \"payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr\" (UID: \"2bab320c-4018-44c6-a7cf-06f76f5f315a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:02.361092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.357719    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:02.568528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.568495    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"]
Oct 31 01:52:02.573991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.573969    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.582251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.582230    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-278a56-openshift-default-dockercfg-mn49h\""
Oct 31 01:52:02.593319 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.593291    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"]
Oct 31 01:52:02.607117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.606432    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781003    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-envoy\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781144    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-credential-socket\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781278    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-socket\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781309    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98gn\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-kube-api-access-t98gn\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781392    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-certs\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781434    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-token\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781618    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f874ec-7f56-4756-8959-fe8913ef679c-istiod-ca-cert\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781653    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-podinfo\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.816067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.781686    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-data\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.825766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.825732    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"]
Oct 31 01:52:02.834893 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:52:02.834839    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bab320c_4018_44c6_a7cf_06f76f5f315a.slice/crio-37f951397d999d36ef855a0b3dcc6a59e8d75e293f5553d4819a1ed8d2faf926 WatchSource:0}: Error finding container 37f951397d999d36ef855a0b3dcc6a59e8d75e293f5553d4819a1ed8d2faf926: Status 404 returned error can't find the container with id 37f951397d999d36ef855a0b3dcc6a59e8d75e293f5553d4819a1ed8d2faf926
Oct 31 01:52:02.938458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.938422    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-credential-socket\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.938739 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.938707    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-socket\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.938887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.938870    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98gn\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-kube-api-access-t98gn\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.939146 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.939113    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-certs\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.939264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.939248    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-token\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.948064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.939555    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f874ec-7f56-4756-8959-fe8913ef679c-istiod-ca-cert\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.948268 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.948247    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-podinfo\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.948393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.948380    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-data\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.948496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.948484    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-envoy\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.958066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.956282    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-certs\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.958066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.956546    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-credential-socket\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.958066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.956775    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-socket\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.958066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.957709    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f874ec-7f56-4756-8959-fe8913ef679c-istiod-ca-cert\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.963479 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.958605    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-data\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.965988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.965964    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-podinfo\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.985790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.985591    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-envoy\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.985790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.985741    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98gn\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-kube-api-access-t98gn\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:02.987786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:02.987634    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-token\") pod \"e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") " pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:03.173243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.173220    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"]
Oct 31 01:52:03.175650 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:52:03.175611    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8351577d_def3_46f5_9ccd_30772fa5a5c9.slice/crio-7d69388aaa0e28c0984484243a42ccac0f266c3f05a67be5d378152fd7d36f54 WatchSource:0}: Error finding container 7d69388aaa0e28c0984484243a42ccac0f266c3f05a67be5d378152fd7d36f54: Status 404 returned error can't find the container with id 7d69388aaa0e28c0984484243a42ccac0f266c3f05a67be5d378152fd7d36f54
Oct 31 01:52:03.189729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.189706    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:03.558371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.558347    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"]
Oct 31 01:52:03.560724 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:52:03.560684    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f874ec_7f56_4756_8959_fe8913ef679c.slice/crio-0a8b99414d9f7542be251255307d86c09b853689bf7e78a2c628b81e44e2a22f WatchSource:0}: Error finding container 0a8b99414d9f7542be251255307d86c09b853689bf7e78a2c628b81e44e2a22f: Status 404 returned error can't find the container with id 0a8b99414d9f7542be251255307d86c09b853689bf7e78a2c628b81e44e2a22f
Oct 31 01:52:03.564063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.564004    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:03.564163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.564114    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:03.564217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.564176    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:03.846241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.846150    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr" event={"ID":"2bab320c-4018-44c6-a7cf-06f76f5f315a","Type":"ContainerStarted","Data":"f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a"}
Oct 31 01:52:03.846241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.846216    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr" event={"ID":"2bab320c-4018-44c6-a7cf-06f76f5f315a","Type":"ContainerStarted","Data":"37f951397d999d36ef855a0b3dcc6a59e8d75e293f5553d4819a1ed8d2faf926"}
Oct 31 01:52:03.846472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.846266    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:03.848271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.848241    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs" event={"ID":"8351577d-def3-46f5-9ccd-30772fa5a5c9","Type":"ContainerStarted","Data":"adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36"}
Oct 31 01:52:03.848413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.848285    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs" event={"ID":"8351577d-def3-46f5-9ccd-30772fa5a5c9","Type":"ContainerStarted","Data":"7d69388aaa0e28c0984484243a42ccac0f266c3f05a67be5d378152fd7d36f54"}
Oct 31 01:52:03.848413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.848330    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:03.850165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.850140    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" event={"ID":"e4f874ec-7f56-4756-8959-fe8913ef679c","Type":"ContainerStarted","Data":"52a3aff7a784858b06f9d82ee96df3fd7d8e1b97502fbb59b8a607e71c206266"}
Oct 31 01:52:03.850278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.850177    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" event={"ID":"e4f874ec-7f56-4756-8959-fe8913ef679c","Type":"ContainerStarted","Data":"0a8b99414d9f7542be251255307d86c09b853689bf7e78a2c628b81e44e2a22f"}
Oct 31 01:52:03.866720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.866674    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr" podStartSLOduration=1.866662518 podStartE2EDuration="1.866662518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:03.862808581 +0000 UTC m=+1881.811456972" watchObservedRunningTime="2026-10-31 01:52:03.866662518 +0000 UTC m=+1881.815310909"
Oct 31 01:52:03.882431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.882391    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs" podStartSLOduration=2.882379369 podStartE2EDuration="2.882379369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:03.880646422 +0000 UTC m=+1881.829294816" watchObservedRunningTime="2026-10-31 01:52:03.882379369 +0000 UTC m=+1881.831027809"
Oct 31 01:52:03.911856 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:03.911780    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podStartSLOduration=1.911760096 podStartE2EDuration="1.911760096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:03.90658267 +0000 UTC m=+1881.855231064" watchObservedRunningTime="2026-10-31 01:52:03.911760096 +0000 UTC m=+1881.860408489"
Oct 31 01:52:04.227064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:04.222299    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:04.316280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:04.316250    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:04.633788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:04.633689    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"]
Oct 31 01:52:04.669105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:04.664610    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c06ae4-d74hl"]
Oct 31 01:52:04.858864 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:04.858830    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:04.872188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:04.872153    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:06.184065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:06.183391    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1511b2-c2be-457d-ac6d-91963d36f8f2" path="/var/lib/kubelet/pods/1f1511b2-c2be-457d-ac6d-91963d36f8f2/volumes"
Oct 31 01:52:08.209252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.209214    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7" podUID="2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" containerName="main" containerID="cri-o://ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78" gracePeriod=45
Oct 31 01:52:08.504064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.504009    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:52:08.623273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.623245    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-home\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-home\") pod \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") "
Oct 31 01:52:08.623464 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.623305    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tls-certs\") pod \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") "
Oct 31 01:52:08.623535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.623488    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-home" pod "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" (UID: "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:08.623593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.623568    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tmp-dir\") pod \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") "
Oct 31 01:52:08.623706 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.623690    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-kube-api-access-ndpss\" (UniqueName: \"kubernetes.io/projected/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-kube-api-access-ndpss\") pod \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") "
Oct 31 01:52:08.623791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.623726    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-model-cache\") pod \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") "
Oct 31 01:52:08.623791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.623783    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tmp-dir" pod "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" (UID: "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:08.623908 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.623811    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-dshm\") pod \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\" (UID: \"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7\") "
Oct 31 01:52:08.624315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.624163    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-model-cache" pod "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" (UID: "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:08.624780 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.624758    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:08.624889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.624785    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:08.624889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.624800    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:08.626215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.626191    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-kube-api-access-ndpss" pod "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" (UID: "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7"). InnerVolumeSpecName "kube-api-access-ndpss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:08.626326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.626222    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tls-certs" pod "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" (UID: "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:08.626369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.626322    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-dshm" pod "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" (UID: "2aa1a07e-2e64-4929-a6c7-a03eecec0cc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:08.725847 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.725773    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndpss\" (UniqueName: \"kubernetes.io/projected/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-kube-api-access-ndpss\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:08.725847 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.725804    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:08.725847 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.725813    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:08.935098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.935059    5293 generic.go:356] "Generic (PLEG): container finished" podID="2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" containerID="ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78" exitCode=0
Oct 31 01:52:08.935310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.935142    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"
Oct 31 01:52:08.935310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.935163    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7" event={"ID":"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7","Type":"ContainerDied","Data":"ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78"}
Oct 31 01:52:08.935310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.935214    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7" event={"ID":"2aa1a07e-2e64-4929-a6c7-a03eecec0cc7","Type":"ContainerDied","Data":"7ee2baec4c11bddfbb548729dc423ccbdb2927a93ed4a4b080a9ab3635f13497"}
Oct 31 01:52:08.935310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.935233    5293 scope.go:122] "RemoveContainer" containerID="ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78"
Oct 31 01:52:08.948266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.948244    5293 scope.go:122] "RemoveContainer" containerID="ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78"
Oct 31 01:52:08.948560 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:52:08.948541    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78\": container with ID starting with ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78 not found: ID does not exist" containerID="ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78"
Oct 31 01:52:08.948610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.948571    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78"} err="failed to get container status \"ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78\": rpc error: code = NotFound desc = could not find container \"ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78\": container with ID starting with ad136a88028d952f2660afb0933e1e27761a2678f7c006dca5238dee0634be78 not found: ID does not exist"
Oct 31 01:52:08.958363 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.958330    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"]
Oct 31 01:52:08.964253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:08.964231    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-03004a/ipp-route-03004a-kserve-8b8c57c85-v4pr7"]
Oct 31 01:52:10.187846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:10.187803    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" path="/var/lib/kubelet/pods/2aa1a07e-2e64-4929-a6c7-a03eecec0cc7/volumes"
Oct 31 01:52:11.982071 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:52:11.979919    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:52:13.041645 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:13.041610    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"]
Oct 31 01:52:13.080061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:13.076487    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd9cdb0-p8f8g"]
Oct 31 01:52:14.176910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:14.176863    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417ee2e3-c6e4-4ba9-933b-d57ba7457454" path="/var/lib/kubelet/pods/417ee2e3-c6e4-4ba9-933b-d57ba7457454/volumes"
Oct 31 01:52:14.860664 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:14.860633    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:52:14.860864 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:14.860746    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:52:15.274808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.274775    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"]
Oct 31 01:52:15.275511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.275495    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" containerName="main"
Oct 31 01:52:15.275567 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.275514    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" containerName="main"
Oct 31 01:52:15.275620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.275611    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa1a07e-2e64-4929-a6c7-a03eecec0cc7" containerName="main"
Oct 31 01:52:15.281173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.281151    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.285604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.285576    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:52:15.285604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.285578    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:52:15.285800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.285616    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:52:15.285800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.285768    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:52:15.288941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.288918    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"]
Oct 31 01:52:15.381695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.381654    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvj7\" (UniqueName: \"kubernetes.io/projected/e74868fb-1470-465e-b771-bc2a7d694a77-kube-api-access-tzvj7\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.381922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.381901    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.382002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.381950    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.482910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.482879    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.483142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.482923    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.483142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.483028    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvj7\" (UniqueName: \"kubernetes.io/projected/e74868fb-1470-465e-b771-bc2a7d694a77-kube-api-access-tzvj7\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.483700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.483680    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.485556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.485533    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.495083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.495027    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvj7\" (UniqueName: \"kubernetes.io/projected/e74868fb-1470-465e-b771-bc2a7d694a77-kube-api-access-tzvj7\") pod \"maas-api-revoke-sub-ccddf611-chwqn\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.601695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.601613    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:15.884190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:15.884158    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"]
Oct 31 01:52:15.885978 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:52:15.885951    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74868fb_1470_465e_b771_bc2a7d694a77.slice/crio-506a951ffeb6f7f33e511d161c8a88f4203312f2cb666b677ca63cdb856c22c5 WatchSource:0}: Error finding container 506a951ffeb6f7f33e511d161c8a88f4203312f2cb666b677ca63cdb856c22c5: Status 404 returned error can't find the container with id 506a951ffeb6f7f33e511d161c8a88f4203312f2cb666b677ca63cdb856c22c5
Oct 31 01:52:16.072876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:16.072840    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn" event={"ID":"e74868fb-1470-465e-b771-bc2a7d694a77","Type":"ContainerStarted","Data":"506a951ffeb6f7f33e511d161c8a88f4203312f2cb666b677ca63cdb856c22c5"}
Oct 31 01:52:16.913443 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:52:16.913407    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74868fb_1470_465e_b771_bc2a7d694a77.slice/crio-5d8a568a8c7146a5ed892d13f0aa6d4dd8e5e9a25b4c2e458c0cc9c23af2e875.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:52:17.087010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:17.086971    5293 generic.go:356] "Generic (PLEG): container finished" podID="e74868fb-1470-465e-b771-bc2a7d694a77" containerID="5d8a568a8c7146a5ed892d13f0aa6d4dd8e5e9a25b4c2e458c0cc9c23af2e875" exitCode=0
Oct 31 01:52:17.087684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:17.087096    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn" event={"ID":"e74868fb-1470-465e-b771-bc2a7d694a77","Type":"ContainerDied","Data":"5d8a568a8c7146a5ed892d13f0aa6d4dd8e5e9a25b4c2e458c0cc9c23af2e875"}
Oct 31 01:52:18.260932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.260903    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:18.301880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.301842    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74868fb-1470-465e-b771-bc2a7d694a77-kube-api-access-tzvj7\" (UniqueName: \"kubernetes.io/projected/e74868fb-1470-465e-b771-bc2a7d694a77-kube-api-access-tzvj7\") pod \"e74868fb-1470-465e-b771-bc2a7d694a77\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") "
Oct 31 01:52:18.302082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.302054    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-openshift-ca\") pod \"e74868fb-1470-465e-b771-bc2a7d694a77\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") "
Oct 31 01:52:18.302180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.302165    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-xks-ca\") pod \"e74868fb-1470-465e-b771-bc2a7d694a77\" (UID: \"e74868fb-1470-465e-b771-bc2a7d694a77\") "
Oct 31 01:52:18.305586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.305446    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74868fb-1470-465e-b771-bc2a7d694a77-kube-api-access-tzvj7" pod "e74868fb-1470-465e-b771-bc2a7d694a77" (UID: "e74868fb-1470-465e-b771-bc2a7d694a77"). InnerVolumeSpecName "kube-api-access-tzvj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:18.306473 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.306440    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-xks-ca" pod "e74868fb-1470-465e-b771-bc2a7d694a77" (UID: "e74868fb-1470-465e-b771-bc2a7d694a77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:18.316054 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.316009    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-openshift-ca" pod "e74868fb-1470-465e-b771-bc2a7d694a77" (UID: "e74868fb-1470-465e-b771-bc2a7d694a77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:18.403872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.403842    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:18.403872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.403870    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74868fb-1470-465e-b771-bc2a7d694a77-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:18.404077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:18.403880    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzvj7\" (UniqueName: \"kubernetes.io/projected/e74868fb-1470-465e-b771-bc2a7d694a77-kube-api-access-tzvj7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:19.108209 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:19.108177    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"
Oct 31 01:52:19.108406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:19.108226    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn" event={"ID":"e74868fb-1470-465e-b771-bc2a7d694a77","Type":"ContainerDied","Data":"506a951ffeb6f7f33e511d161c8a88f4203312f2cb666b677ca63cdb856c22c5"}
Oct 31 01:52:19.108406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:19.108281    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="506a951ffeb6f7f33e511d161c8a88f4203312f2cb666b677ca63cdb856c22c5"
Oct 31 01:52:20.163576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.163533    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"]
Oct 31 01:52:20.164894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.164866    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e74868fb-1470-465e-b771-bc2a7d694a77" containerName="revoke-keys"
Oct 31 01:52:20.164894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.164892    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74868fb-1470-465e-b771-bc2a7d694a77" containerName="revoke-keys"
Oct 31 01:52:20.165176 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.165159    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74868fb-1470-465e-b771-bc2a7d694a77" containerName="revoke-keys"
Oct 31 01:52:20.172166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.172148    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"]
Oct 31 01:52:20.172278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.172265    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.179124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.179087    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-925bac-openshift-default-dockercfg-tr4kp\""
Oct 31 01:52:20.251333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251295    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.251509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251356    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.251509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251427    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4n9\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-kube-api-access-fs4n9\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.251509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251462    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.251640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251520    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.251687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251653    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.251734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251688    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.251734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251720    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.251894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.251878    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.354957    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.355111    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.355222    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.355313    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.355338    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4n9\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-kube-api-access-fs4n9\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.355365    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.355442    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.355533    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.356064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.355556    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.357524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.357494    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.378827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.358023    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.379016 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.358646    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.379136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.358882    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.379243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.374379    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.382392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.382324    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.391063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.386212    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.401089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.400728    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4n9\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-kube-api-access-fs4n9\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.431709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.430244    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.493770 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.493725    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:20.877242 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.877216    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"]
Oct 31 01:52:20.878186 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:52:20.878164    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c3959c_9fd5_4a05_b004_a27bab56e7a5.slice/crio-ddf43563679b7953cce8e21bee1a851ec18e2df6e29d8f7d97ee8e7932aa0da4 WatchSource:0}: Error finding container ddf43563679b7953cce8e21bee1a851ec18e2df6e29d8f7d97ee8e7932aa0da4: Status 404 returned error can't find the container with id ddf43563679b7953cce8e21bee1a851ec18e2df6e29d8f7d97ee8e7932aa0da4
Oct 31 01:52:20.880793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.880758    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:20.880901 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.880839    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:20.880901 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:20.880887    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:21.159703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:21.159667    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" event={"ID":"a1c3959c-9fd5-4a05-b004-a27bab56e7a5","Type":"ContainerStarted","Data":"9a2f37bc55d219e810735f18858dd17a05c378eca95482c3b29686ec9f48cd81"}
Oct 31 01:52:21.159885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:21.159719    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" event={"ID":"a1c3959c-9fd5-4a05-b004-a27bab56e7a5","Type":"ContainerStarted","Data":"ddf43563679b7953cce8e21bee1a851ec18e2df6e29d8f7d97ee8e7932aa0da4"}
Oct 31 01:52:21.184313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:21.184256    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podStartSLOduration=1.184243833 podStartE2EDuration="1.184243833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:21.181986121 +0000 UTC m=+1899.130634514" watchObservedRunningTime="2026-10-31 01:52:21.184243833 +0000 UTC m=+1899.132892224"
Oct 31 01:52:21.494905 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:21.494820    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:22.501034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:22.500956    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:23.174378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:23.174352    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:23.175269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:23.175247    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:52:25.944478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:25.944430    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:52:28.277244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.276480    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"]
Oct 31 01:52:28.294012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.293986    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"]
Oct 31 01:52:28.294296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.294281    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.301976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.301947    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:52:28.308294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.308275    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:52:28.308552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.308537    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:52:28.310191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.310160    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:52:28.479152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.479098    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490c0eee-620c-4051-b61e-973983471135-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.479686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.479647    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mh6b\" (UniqueName: \"kubernetes.io/projected/490c0eee-620c-4051-b61e-973983471135-kube-api-access-6mh6b\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.479865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.479843    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490c0eee-620c-4051-b61e-973983471135-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.600123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.593679    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mh6b\" (UniqueName: \"kubernetes.io/projected/490c0eee-620c-4051-b61e-973983471135-kube-api-access-6mh6b\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.600123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.593741    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490c0eee-620c-4051-b61e-973983471135-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.600123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.593877    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490c0eee-620c-4051-b61e-973983471135-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.622075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.620783    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490c0eee-620c-4051-b61e-973983471135-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.661066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.657103    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mh6b\" (UniqueName: \"kubernetes.io/projected/490c0eee-620c-4051-b61e-973983471135-kube-api-access-6mh6b\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.672218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.672189    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490c0eee-620c-4051-b61e-973983471135-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:28.768101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:28.759821    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:29.757389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:29.757347    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"]
Oct 31 01:52:30.428445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.428412    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"]
Oct 31 01:52:30.456180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.456137    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:52:30.480904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.474809    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:52:30.548678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.548647    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"]
Oct 31 01:52:30.549092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.549074    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.697123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.694631    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmwvq\" (UniqueName: \"kubernetes.io/projected/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-kube-api-access-mmwvq\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.708060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.698105    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.708060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.698307    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.782786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.782756    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv" event={"ID":"490c0eee-620c-4051-b61e-973983471135","Type":"ContainerStarted","Data":"bafcf388c8b7d0a8dfd21f97efba83f98f466b97e0d40b85e52985542003674a"}
Oct 31 01:52:30.821535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.821215    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmwvq\" (UniqueName: \"kubernetes.io/projected/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-kube-api-access-mmwvq\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.821535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.821269    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.821535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.821374    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.823347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.822936    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.865345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.865314    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:30.866428 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:30.866408    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmwvq\" (UniqueName: \"kubernetes.io/projected/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-kube-api-access-mmwvq\") pod \"maas-api-revoke-sub-5c678dcc-8rgx5\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:31.068361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:31.068321    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:31.569850 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:31.569812    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"]
Oct 31 01:52:31.883512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:31.883476    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5" event={"ID":"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50","Type":"ContainerStarted","Data":"ef7aacd798537c7dedb4c358af0d9139ab27b3d7f366d2f225ccc20e5806a4fd"}
Oct 31 01:52:31.888606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:31.888573    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv" event={"ID":"490c0eee-620c-4051-b61e-973983471135","Type":"ContainerStarted","Data":"6525c8fedfa986088a6806e97312e9bed809725794b4611f4fac4da7d81fb715"}
Oct 31 01:52:32.910077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:32.902812    5293 generic.go:356] "Generic (PLEG): container finished" podID="88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50" containerID="63661078ffaf934299db38c08232c91d2566a39687590edea44c6d36ef8a1a24" exitCode=0
Oct 31 01:52:32.910077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:32.903022    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5" event={"ID":"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50","Type":"ContainerDied","Data":"63661078ffaf934299db38c08232c91d2566a39687590edea44c6d36ef8a1a24"}
Oct 31 01:52:32.910077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:32.909254    5293 generic.go:356] "Generic (PLEG): container finished" podID="490c0eee-620c-4051-b61e-973983471135" containerID="6525c8fedfa986088a6806e97312e9bed809725794b4611f4fac4da7d81fb715" exitCode=0
Oct 31 01:52:32.910077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:32.909407    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv" event={"ID":"490c0eee-620c-4051-b61e-973983471135","Type":"ContainerDied","Data":"6525c8fedfa986088a6806e97312e9bed809725794b4611f4fac4da7d81fb715"}
Oct 31 01:52:33.279837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.279534    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:33.366696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.366610    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490c0eee-620c-4051-b61e-973983471135-kube-api-access-6mh6b\" (UniqueName: \"kubernetes.io/projected/490c0eee-620c-4051-b61e-973983471135-kube-api-access-6mh6b\") pod \"490c0eee-620c-4051-b61e-973983471135\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") "
Oct 31 01:52:33.379406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.379368    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490c0eee-620c-4051-b61e-973983471135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490c0eee-620c-4051-b61e-973983471135-maas-api-openshift-ca\") pod \"490c0eee-620c-4051-b61e-973983471135\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") "
Oct 31 01:52:33.379735 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.379715    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/490c0eee-620c-4051-b61e-973983471135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490c0eee-620c-4051-b61e-973983471135-maas-api-xks-ca\") pod \"490c0eee-620c-4051-b61e-973983471135\" (UID: \"490c0eee-620c-4051-b61e-973983471135\") "
Oct 31 01:52:33.454104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.411695    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490c0eee-620c-4051-b61e-973983471135-maas-api-openshift-ca" pod "490c0eee-620c-4051-b61e-973983471135" (UID: "490c0eee-620c-4051-b61e-973983471135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:33.535289 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.505588    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490c0eee-620c-4051-b61e-973983471135-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:33.547733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.547675    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490c0eee-620c-4051-b61e-973983471135-kube-api-access-6mh6b" pod "490c0eee-620c-4051-b61e-973983471135" (UID: "490c0eee-620c-4051-b61e-973983471135"). InnerVolumeSpecName "kube-api-access-6mh6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:33.566352 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.565020    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490c0eee-620c-4051-b61e-973983471135-maas-api-xks-ca" pod "490c0eee-620c-4051-b61e-973983471135" (UID: "490c0eee-620c-4051-b61e-973983471135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:33.617363 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.617263    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mh6b\" (UniqueName: \"kubernetes.io/projected/490c0eee-620c-4051-b61e-973983471135-kube-api-access-6mh6b\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:33.617655 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:33.617635    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490c0eee-620c-4051-b61e-973983471135-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:34.027004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.024895    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"
Oct 31 01:52:34.028141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.028104    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv" event={"ID":"490c0eee-620c-4051-b61e-973983471135","Type":"ContainerDied","Data":"bafcf388c8b7d0a8dfd21f97efba83f98f466b97e0d40b85e52985542003674a"}
Oct 31 01:52:34.029083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.029058    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bafcf388c8b7d0a8dfd21f97efba83f98f466b97e0d40b85e52985542003674a"
Oct 31 01:52:34.687309 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.687237    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:34.791988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.791956    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-kube-api-access-mmwvq\" (UniqueName: \"kubernetes.io/projected/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-kube-api-access-mmwvq\") pod \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") "
Oct 31 01:52:34.792186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.792162    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-xks-ca\") pod \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") "
Oct 31 01:52:34.792232 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.792218    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-openshift-ca\") pod \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\" (UID: \"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50\") "
Oct 31 01:52:34.792713 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.792673    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-openshift-ca" pod "88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50" (UID: "88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:34.794671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.794653    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-kube-api-access-mmwvq" pod "88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50" (UID: "88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50"). InnerVolumeSpecName "kube-api-access-mmwvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:34.794811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.794735    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-xks-ca" pod "88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50" (UID: "88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:34.894491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.894451    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:34.894491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.894484    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:34.894692 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:34.894499    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmwvq\" (UniqueName: \"kubernetes.io/projected/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50-kube-api-access-mmwvq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:35.089114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:35.086950    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5" event={"ID":"88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50","Type":"ContainerDied","Data":"ef7aacd798537c7dedb4c358af0d9139ab27b3d7f366d2f225ccc20e5806a4fd"}
Oct 31 01:52:35.089114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:35.087073    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef7aacd798537c7dedb4c358af0d9139ab27b3d7f366d2f225ccc20e5806a4fd"
Oct 31 01:52:35.089114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:35.087164    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"
Oct 31 01:52:39.595429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:39.593569    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": read tcp 10.133.0.2:47026->10.133.0.175:15021: read: connection reset by peer" start-of-body=
Oct 31 01:52:39.595429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:39.593628    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": read tcp 10.133.0.2:47026->10.133.0.175:15021: read: connection reset by peer"
Oct 31 01:52:40.024060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.019188    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"]
Oct 31 01:52:40.051541 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.051507    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b8d890-b2lpv"]
Oct 31 01:52:40.154024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.153677    5293 generic.go:356] "Generic (PLEG): container finished" podID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerID="52a3aff7a784858b06f9d82ee96df3fd7d8e1b97502fbb59b8a607e71c206266" exitCode=0
Oct 31 01:52:40.154024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.153709    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" event={"ID":"e4f874ec-7f56-4756-8959-fe8913ef679c","Type":"ContainerDied","Data":"52a3aff7a784858b06f9d82ee96df3fd7d8e1b97502fbb59b8a607e71c206266"}
Oct 31 01:52:40.154645 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.154623    5293 scope.go:122] "RemoveContainer" containerID="52a3aff7a784858b06f9d82ee96df3fd7d8e1b97502fbb59b8a607e71c206266"
Oct 31 01:52:40.157497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.157333    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:40.157497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.157419    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:40.157497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.157467    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:40.159352 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.159274    5293 generic.go:356] "Generic (PLEG): container finished" podID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerID="2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480" exitCode=0
Oct 31 01:52:40.159450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.159389    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" event={"ID":"b8234a61-a840-4b87-b7d0-0868567a98e3","Type":"ContainerDied","Data":"2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480"}
Oct 31 01:52:40.161101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.159847    5293 scope.go:122] "RemoveContainer" containerID="2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480"
Oct 31 01:52:40.174117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.174027    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:40.174240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.174180    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:40.174304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.174236    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:40.179377 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:40.179252    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490c0eee-620c-4051-b61e-973983471135" path="/var/lib/kubelet/pods/490c0eee-620c-4051-b61e-973983471135/volumes"
Oct 31 01:52:41.170657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:41.170630    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" event={"ID":"e4f874ec-7f56-4756-8959-fe8913ef679c","Type":"ContainerStarted","Data":"8a77c40754a1f835eb94a54a4b38f7c7eb39d9bf4f087f3701c4e8c957c15138"}
Oct 31 01:52:41.172935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:41.172909    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" event={"ID":"b8234a61-a840-4b87-b7d0-0868567a98e3","Type":"ContainerStarted","Data":"9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616"}
Oct 31 01:52:41.190458 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:41.190440    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:41.190684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:41.190663    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 01:52:41.190783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:41.190714    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 01:52:41.339181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:41.337731    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:52:41.339181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:41.338663    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:52:41.339181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:41.338726    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 01:52:41.789101 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:52:41.789033    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:52:42.190861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.190827    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 01:52:42.191364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.190873    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 01:52:42.347235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.347197    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:52:42.347510 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.347478    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 01:52:42.958916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.958874    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"]
Oct 31 01:52:42.959798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.959772    5293 cpu_manager.go:414] "containerMap: removing container" podUID="490c0eee-620c-4051-b61e-973983471135" containerName="revoke-keys"
Oct 31 01:52:42.959798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.959794    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490c0eee-620c-4051-b61e-973983471135" containerName="revoke-keys"
Oct 31 01:52:42.960010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.959815    5293 cpu_manager.go:414] "containerMap: removing container" podUID="88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50" containerName="revoke-keys"
Oct 31 01:52:42.960010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.959820    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50" containerName="revoke-keys"
Oct 31 01:52:42.960010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.959960    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="490c0eee-620c-4051-b61e-973983471135" containerName="revoke-keys"
Oct 31 01:52:42.960010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.959970    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50" containerName="revoke-keys"
Oct 31 01:52:42.963488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.963463    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:42.967483 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.967464    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:52:42.967767 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.967752    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:52:42.968205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.968187    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:52:42.968608 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.968587    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:52:42.970109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:42.970087    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"]
Oct 31 01:52:43.065261 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.065221    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m287q\" (UniqueName: \"kubernetes.io/projected/e9d16eaa-b93b-4279-aea4-2add95143e7d-kube-api-access-m287q\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.065811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.065788    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.066025 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.066007    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.167606 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.167573    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m287q\" (UniqueName: \"kubernetes.io/projected/e9d16eaa-b93b-4279-aea4-2add95143e7d-kube-api-access-m287q\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.167849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.167818    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.167920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.167898    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.168670 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.168647    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.171122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.170587    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.187166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.187141    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"]
Oct 31 01:52:43.191128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.191050    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 01:52:43.191420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.191134    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 01:52:43.193879 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.193451    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.195681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.195650    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m287q\" (UniqueName: \"kubernetes.io/projected/e9d16eaa-b93b-4279-aea4-2add95143e7d-kube-api-access-m287q\") pod \"maas-api-revoke-keys-e2e-ipp-b-40a891-n827s\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.225113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.224303    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"]
Oct 31 01:52:43.287716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.287689    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:43.332288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.331492    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:52:43.332288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.331556    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 01:52:43.376970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.376918    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.377096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.376972    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrlx\" (UniqueName: \"kubernetes.io/projected/c90b9e1a-8690-40a4-8279-49c974f5b666-kube-api-access-lxrlx\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.377148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.377092    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.479977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.479230    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.479977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.479283    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrlx\" (UniqueName: \"kubernetes.io/projected/c90b9e1a-8690-40a4-8279-49c974f5b666-kube-api-access-lxrlx\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.479977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.479371    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.480471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.480441    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.483298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.483274    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.758852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.758739    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrlx\" (UniqueName: \"kubernetes.io/projected/c90b9e1a-8690-40a4-8279-49c974f5b666-kube-api-access-lxrlx\") pod \"maas-api-revoke-sub-212cd153-92fwz\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.889743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.889710    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:43.975379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:43.975350    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"]
Oct 31 01:52:44.191414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.191379    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 01:52:44.192098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.192063    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 01:52:44.278604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.278521    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s" event={"ID":"e9d16eaa-b93b-4279-aea4-2add95143e7d","Type":"ContainerStarted","Data":"b6e26b2a401e69f4d5177032c76705880cf1557d8d2bac0c6cc3e97b8c9ca0c9"}
Oct 31 01:52:44.327065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.326088    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"]
Oct 31 01:52:44.342074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.336606    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:52:44.342074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.336664    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 01:52:44.422087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.408061    5293 scope.go:122] "RemoveContainer" containerID="6fa4ec8dd7bf804ff131444146ebf5d43cc9212e32490fc00bb496d4f4d42ee5"
Oct 31 01:52:44.576120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.576090    5293 scope.go:122] "RemoveContainer" containerID="64a298fd66765d907ab07b1b290da74b7b7f048946cd250c11b5b31d9b62a6d4"
Oct 31 01:52:44.654858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.654833    5293 scope.go:122] "RemoveContainer" containerID="bc67b856ec426647ef166b2d808a70acb21ac1f2a8745040b9c4038e8554dbbf"
Oct 31 01:52:44.668407 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:44.668384    5293 scope.go:122] "RemoveContainer" containerID="c562ce3a0577aa6ee108c6f552aee2bc5a82db4b63de2babad54782d4a5acc3a"
Oct 31 01:52:45.137106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.137009    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"]
Oct 31 01:52:45.178065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.175270    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa1d0d6-glq9v"]
Oct 31 01:52:45.200969 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.199100    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 01:52:45.200969 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.199184    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 01:52:45.343068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.341267    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:52:45.343068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.341501    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 01:52:45.442106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.437683    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s" event={"ID":"e9d16eaa-b93b-4279-aea4-2add95143e7d","Type":"ContainerStarted","Data":"4934ac4d24165d5e771b60298caa80a08b56a3defba2145e375b28eecf06d7b6"}
Oct 31 01:52:45.469121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.468317    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s" podStartSLOduration=3.468297545 podStartE2EDuration="3.468297545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:45.46657846 +0000 UTC m=+1923.415226853" watchObservedRunningTime="2026-10-31 01:52:45.468297545 +0000 UTC m=+1923.416945940"
Oct 31 01:52:45.471216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.471183    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz" event={"ID":"c90b9e1a-8690-40a4-8279-49c974f5b666","Type":"ContainerStarted","Data":"d355cbc0cd448279bd563e64950e385249d75e354c634eb849ecb36382ea0a69"}
Oct 31 01:52:45.471438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.471235    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz" event={"ID":"c90b9e1a-8690-40a4-8279-49c974f5b666","Type":"ContainerStarted","Data":"2ed731739a00d3cf1fec59b9171b771b54dea89e5931045e93d22a32466b0afc"}
Oct 31 01:52:45.495507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:45.495457    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz" podStartSLOduration=2.495441267 podStartE2EDuration="2.495441267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:45.490919132 +0000 UTC m=+1923.439567523" watchObservedRunningTime="2026-10-31 01:52:45.495441267 +0000 UTC m=+1923.444089681"
Oct 31 01:52:46.193115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.192314    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 01:52:46.193115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.192403    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 01:52:46.239066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.238023    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb13ac4-8830-478d-a100-b4a432b0c702" path="/var/lib/kubelet/pods/3cb13ac4-8830-478d-a100-b4a432b0c702/volumes"
Oct 31 01:52:46.340019 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.339570    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:52:46.340019 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.339629    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 01:52:46.480982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.480900    5293 generic.go:356] "Generic (PLEG): container finished" podID="e9d16eaa-b93b-4279-aea4-2add95143e7d" containerID="4934ac4d24165d5e771b60298caa80a08b56a3defba2145e375b28eecf06d7b6" exitCode=0
Oct 31 01:52:46.481158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.480982    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s" event={"ID":"e9d16eaa-b93b-4279-aea4-2add95143e7d","Type":"ContainerDied","Data":"4934ac4d24165d5e771b60298caa80a08b56a3defba2145e375b28eecf06d7b6"}
Oct 31 01:52:46.482562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.482539    5293 generic.go:356] "Generic (PLEG): container finished" podID="c90b9e1a-8690-40a4-8279-49c974f5b666" containerID="d355cbc0cd448279bd563e64950e385249d75e354c634eb849ecb36382ea0a69" exitCode=0
Oct 31 01:52:46.482744 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:46.482729    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz" event={"ID":"c90b9e1a-8690-40a4-8279-49c974f5b666","Type":"ContainerDied","Data":"d355cbc0cd448279bd563e64950e385249d75e354c634eb849ecb36382ea0a69"}
Oct 31 01:52:47.196207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:47.196178    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:47.196652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:47.196630    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:47.197398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:47.197379    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:52:47.335732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:47.335702    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:52:47.336199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:47.336141    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:52:47.336832 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:47.336809    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:52:47.967785 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:47.967733    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"]
Oct 31 01:52:48.011553 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.011424    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:48.015511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.015479    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:48.080697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.080662    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-openshift-ca\") pod \"e9d16eaa-b93b-4279-aea4-2add95143e7d\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") "
Oct 31 01:52:48.080942 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.080919    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-openshift-ca\") pod \"c90b9e1a-8690-40a4-8279-49c974f5b666\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") "
Oct 31 01:52:48.081126 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.081108    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90b9e1a-8690-40a4-8279-49c974f5b666-kube-api-access-lxrlx\" (UniqueName: \"kubernetes.io/projected/c90b9e1a-8690-40a4-8279-49c974f5b666-kube-api-access-lxrlx\") pod \"c90b9e1a-8690-40a4-8279-49c974f5b666\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") "
Oct 31 01:52:48.081321 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.081302    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-xks-ca\") pod \"c90b9e1a-8690-40a4-8279-49c974f5b666\" (UID: \"c90b9e1a-8690-40a4-8279-49c974f5b666\") "
Oct 31 01:52:48.081463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.081448    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d16eaa-b93b-4279-aea4-2add95143e7d-kube-api-access-m287q\" (UniqueName: \"kubernetes.io/projected/e9d16eaa-b93b-4279-aea4-2add95143e7d-kube-api-access-m287q\") pod \"e9d16eaa-b93b-4279-aea4-2add95143e7d\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") "
Oct 31 01:52:48.081714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.081699    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-xks-ca\") pod \"e9d16eaa-b93b-4279-aea4-2add95143e7d\" (UID: \"e9d16eaa-b93b-4279-aea4-2add95143e7d\") "
Oct 31 01:52:48.100484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.100449    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-openshift-ca" pod "e9d16eaa-b93b-4279-aea4-2add95143e7d" (UID: "e9d16eaa-b93b-4279-aea4-2add95143e7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:48.101000 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.100978    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-openshift-ca" pod "c90b9e1a-8690-40a4-8279-49c974f5b666" (UID: "c90b9e1a-8690-40a4-8279-49c974f5b666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:48.146519 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.146463    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d16eaa-b93b-4279-aea4-2add95143e7d-kube-api-access-m287q" pod "e9d16eaa-b93b-4279-aea4-2add95143e7d" (UID: "e9d16eaa-b93b-4279-aea4-2add95143e7d"). InnerVolumeSpecName "kube-api-access-m287q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:48.155776 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.155728    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-xks-ca" pod "e9d16eaa-b93b-4279-aea4-2add95143e7d" (UID: "e9d16eaa-b93b-4279-aea4-2add95143e7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:48.155932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.155744    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90b9e1a-8690-40a4-8279-49c974f5b666-kube-api-access-lxrlx" pod "c90b9e1a-8690-40a4-8279-49c974f5b666" (UID: "c90b9e1a-8690-40a4-8279-49c974f5b666"). InnerVolumeSpecName "kube-api-access-lxrlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:48.160286 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.160258    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-xks-ca" pod "c90b9e1a-8690-40a4-8279-49c974f5b666" (UID: "c90b9e1a-8690-40a4-8279-49c974f5b666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:48.202066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.197211    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:48.202066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.197250    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:48.202066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.197264    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxrlx\" (UniqueName: \"kubernetes.io/projected/c90b9e1a-8690-40a4-8279-49c974f5b666-kube-api-access-lxrlx\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:48.202066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.197280    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90b9e1a-8690-40a4-8279-49c974f5b666-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:48.202066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.197296    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m287q\" (UniqueName: \"kubernetes.io/projected/e9d16eaa-b93b-4279-aea4-2add95143e7d-kube-api-access-m287q\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:48.202066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.197321    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d16eaa-b93b-4279-aea4-2add95143e7d-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:48.475778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.475742    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"]
Oct 31 01:52:48.476405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.476390    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c90b9e1a-8690-40a4-8279-49c974f5b666" containerName="revoke-keys"
Oct 31 01:52:48.476466 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.476406    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90b9e1a-8690-40a4-8279-49c974f5b666" containerName="revoke-keys"
Oct 31 01:52:48.476466 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.476420    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e9d16eaa-b93b-4279-aea4-2add95143e7d" containerName="revoke-keys"
Oct 31 01:52:48.476466 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.476424    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d16eaa-b93b-4279-aea4-2add95143e7d" containerName="revoke-keys"
Oct 31 01:52:48.476565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.476527    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90b9e1a-8690-40a4-8279-49c974f5b666" containerName="revoke-keys"
Oct 31 01:52:48.476565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.476542    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d16eaa-b93b-4279-aea4-2add95143e7d" containerName="revoke-keys"
Oct 31 01:52:48.481015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.480993    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.497403 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.496786    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"]
Oct 31 01:52:48.543164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.543135    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"
Oct 31 01:52:48.543164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.543146    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz" event={"ID":"c90b9e1a-8690-40a4-8279-49c974f5b666","Type":"ContainerDied","Data":"2ed731739a00d3cf1fec59b9171b771b54dea89e5931045e93d22a32466b0afc"}
Oct 31 01:52:48.543412 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.543209    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ed731739a00d3cf1fec59b9171b771b54dea89e5931045e93d22a32466b0afc"
Oct 31 01:52:48.545628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.545606    5293 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" secret="" err="secret \"e2e-praxis-a-b8d890-openshift-default-dockercfg-fvs7g\" not found"
Oct 31 01:52:48.545858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.545835    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"
Oct 31 01:52:48.546685 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.546656    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s" event={"ID":"e9d16eaa-b93b-4279-aea4-2add95143e7d","Type":"ContainerDied","Data":"b6e26b2a401e69f4d5177032c76705880cf1557d8d2bac0c6cc3e97b8c9ca0c9"}
Oct 31 01:52:48.546789 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.546706    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e26b2a401e69f4d5177032c76705880cf1557d8d2bac0c6cc3e97b8c9ca0c9"
Oct 31 01:52:48.552081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.552025    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.552185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.552165    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzb2\" (UniqueName: \"kubernetes.io/projected/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-kube-api-access-lfzb2\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.552243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.552227    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.653720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.653687    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.653936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.653896    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzb2\" (UniqueName: \"kubernetes.io/projected/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-kube-api-access-lfzb2\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.654055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.654017    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.654429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.654406    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.656397 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.656377    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.665607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.665587    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzb2\" (UniqueName: \"kubernetes.io/projected/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-kube-api-access-lfzb2\") pod \"maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:48.805399 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:48.805309    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:49.100274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:49.100247    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"]
Oct 31 01:52:49.101517 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:52:49.101488    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49e4a26_0b38_4fd9_8b1d_bbd818f2c2dc.slice/crio-35d590d0fdcc50b26d6ab4f358ed109b65f4cbaed0789eb586d017617ec42a5a WatchSource:0}: Error finding container 35d590d0fdcc50b26d6ab4f358ed109b65f4cbaed0789eb586d017617ec42a5a: Status 404 returned error can't find the container with id 35d590d0fdcc50b26d6ab4f358ed109b65f4cbaed0789eb586d017617ec42a5a
Oct 31 01:52:49.556267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:49.556227    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd" event={"ID":"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc","Type":"ContainerStarted","Data":"067c6e241f9ccff770642c4763351c226da5bbeeab2c6e340d886034bc4b0400"}
Oct 31 01:52:49.556707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:49.556324    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd" event={"ID":"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc","Type":"ContainerStarted","Data":"35d590d0fdcc50b26d6ab4f358ed109b65f4cbaed0789eb586d017617ec42a5a"}
Oct 31 01:52:49.556707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:49.556345    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy" containerID="cri-o://9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616" gracePeriod=30
Oct 31 01:52:49.595114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:49.595057    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd" podStartSLOduration=1.59502214 podStartE2EDuration="1.59502214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:49.59428296 +0000 UTC m=+1927.542931352" watchObservedRunningTime="2026-10-31 01:52:49.59502214 +0000 UTC m=+1927.543670535"
Oct 31 01:52:50.565932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:50.565839    5293 generic.go:356] "Generic (PLEG): container finished" podID="b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc" containerID="067c6e241f9ccff770642c4763351c226da5bbeeab2c6e340d886034bc4b0400" exitCode=0
Oct 31 01:52:50.566355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:50.565918    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd" event={"ID":"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc","Type":"ContainerDied","Data":"067c6e241f9ccff770642c4763351c226da5bbeeab2c6e340d886034bc4b0400"}
Oct 31 01:52:51.743300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.743270    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:51.848323 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.848239    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-openshift-ca\") pod \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") "
Oct 31 01:52:51.848323 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.848321    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-xks-ca\") pod \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") "
Oct 31 01:52:51.848528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.848462    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-kube-api-access-lfzb2\" (UniqueName: \"kubernetes.io/projected/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-kube-api-access-lfzb2\") pod \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\" (UID: \"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc\") "
Oct 31 01:52:51.848616 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.848593    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-openshift-ca" pod "b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc" (UID: "b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:51.848951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.848932    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:51.850648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.850620    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-xks-ca" pod "b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc" (UID: "b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:51.850782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.850762    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-kube-api-access-lfzb2" pod "b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc" (UID: "b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc"). InnerVolumeSpecName "kube-api-access-lfzb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:51.950288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.950247    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:51.950288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:51.950280    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfzb2\" (UniqueName: \"kubernetes.io/projected/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc-kube-api-access-lfzb2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:52.582942 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:52.582860    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"
Oct 31 01:52:52.582942 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:52.582895    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd" event={"ID":"b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc","Type":"ContainerDied","Data":"35d590d0fdcc50b26d6ab4f358ed109b65f4cbaed0789eb586d017617ec42a5a"}
Oct 31 01:52:52.583130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:52.582972    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d590d0fdcc50b26d6ab4f358ed109b65f4cbaed0789eb586d017617ec42a5a"
Oct 31 01:52:54.401395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.401364    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"]
Oct 31 01:52:54.402342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.401611    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82" podUID="8eb20b7d-f1b3-4773-be3b-335f1271d3a6" containerName="payload-processing" containerID="cri-o://11cbcaeebf2435f992f1d773df32b4df964527421291d02652a9809a361d368a" gracePeriod=30
Oct 31 01:52:54.415362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.415139    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"]
Oct 31 01:52:54.415493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.415393    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4" podUID="a6dedf1a-4f23-403e-b2ce-0d1d1732884a" containerName="payload-pre-processing" containerID="cri-o://2cf74b8e731308d41198e136adc525b8a1be943a7436d7bc9ee826ea2544af62" gracePeriod=30
Oct 31 01:52:54.808423 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.808387    5293 generic.go:356] "Generic (PLEG): container finished" podID="8eb20b7d-f1b3-4773-be3b-335f1271d3a6" containerID="11cbcaeebf2435f992f1d773df32b4df964527421291d02652a9809a361d368a" exitCode=0
Oct 31 01:52:54.821441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.812195    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82" event={"ID":"8eb20b7d-f1b3-4773-be3b-335f1271d3a6","Type":"ContainerDied","Data":"11cbcaeebf2435f992f1d773df32b4df964527421291d02652a9809a361d368a"}
Oct 31 01:52:54.873129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.873095    5293 generic.go:356] "Generic (PLEG): container finished" podID="a6dedf1a-4f23-403e-b2ce-0d1d1732884a" containerID="2cf74b8e731308d41198e136adc525b8a1be943a7436d7bc9ee826ea2544af62" exitCode=0
Oct 31 01:52:54.884067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.873492    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4" event={"ID":"a6dedf1a-4f23-403e-b2ce-0d1d1732884a","Type":"ContainerDied","Data":"2cf74b8e731308d41198e136adc525b8a1be943a7436d7bc9ee826ea2544af62"}
Oct 31 01:52:54.908622 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.908534    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"]
Oct 31 01:52:54.908862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:54.908845    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40a891-n827s"]
Oct 31 01:52:55.095067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.094190    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:52:55.148067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.147738    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-plugins-config-volume\") pod \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\" (UID: \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\") "
Oct 31 01:52:55.148067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.147899    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-kube-api-access-fvb5b\" (UniqueName: \"kubernetes.io/projected/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-kube-api-access-fvb5b\") pod \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\" (UID: \"8eb20b7d-f1b3-4773-be3b-335f1271d3a6\") "
Oct 31 01:52:55.150029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.149303    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-plugins-config-volume" pod "8eb20b7d-f1b3-4773-be3b-335f1271d3a6" (UID: "8eb20b7d-f1b3-4773-be3b-335f1271d3a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:55.153967 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.151661    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-kube-api-access-fvb5b" pod "8eb20b7d-f1b3-4773-be3b-335f1271d3a6" (UID: "8eb20b7d-f1b3-4773-be3b-335f1271d3a6"). InnerVolumeSpecName "kube-api-access-fvb5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:55.195873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.195851    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:52:55.250970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.250935    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-plugins-config-volume\") pod \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\" (UID: \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\") "
Oct 31 01:52:55.251524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.251474    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-kube-api-access-jwdtg\" (UniqueName: \"kubernetes.io/projected/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-kube-api-access-jwdtg\") pod \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\" (UID: \"a6dedf1a-4f23-403e-b2ce-0d1d1732884a\") "
Oct 31 01:52:55.251669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.251536    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-plugins-config-volume" pod "a6dedf1a-4f23-403e-b2ce-0d1d1732884a" (UID: "a6dedf1a-4f23-403e-b2ce-0d1d1732884a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:55.252544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.252518    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.252671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.252550    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.252671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.252568    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvb5b\" (UniqueName: \"kubernetes.io/projected/8eb20b7d-f1b3-4773-be3b-335f1271d3a6-kube-api-access-fvb5b\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.255592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.255562    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-kube-api-access-jwdtg" pod "a6dedf1a-4f23-403e-b2ce-0d1d1732884a" (UID: "a6dedf1a-4f23-403e-b2ce-0d1d1732884a"). InnerVolumeSpecName "kube-api-access-jwdtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:55.310135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.310106    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:52:55.354086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354016    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8234a61-a840-4b87-b7d0-0868567a98e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8234a61-a840-4b87-b7d0-0868567a98e3-istiod-ca-cert\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.355279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354332    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-data\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.355279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354394    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-podinfo\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.355279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354426    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-credential-socket\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.355279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354550    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-token\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-token\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.355279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354618    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-kube-api-access-9bj84\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-kube-api-access-9bj84\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.355279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354674    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-certs\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.355279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354782    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-envoy\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.355279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.354973    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-socket\") pod \"b8234a61-a840-4b87-b7d0-0868567a98e3\" (UID: \"b8234a61-a840-4b87-b7d0-0868567a98e3\") "
Oct 31 01:52:55.356895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.356646    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdtg\" (UniqueName: \"kubernetes.io/projected/a6dedf1a-4f23-403e-b2ce-0d1d1732884a-kube-api-access-jwdtg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.357332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.357262    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-socket" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:55.357430 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.357389    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-certs" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:55.357778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.357613    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-data" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:55.358064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.357977    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-credential-socket" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:55.358064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.357983    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8234a61-a840-4b87-b7d0-0868567a98e3-istiod-ca-cert" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:55.410420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.406612    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-token" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:55.410912 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.410706    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-envoy" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:55.410912 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.410892    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-kube-api-access-9bj84" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "kube-api-access-9bj84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:55.430638 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.430600    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-podinfo" pod "b8234a61-a840-4b87-b7d0-0868567a98e3" (UID: "b8234a61-a840-4b87-b7d0-0868567a98e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:52:55.457988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.457920    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bj84\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-kube-api-access-9bj84\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.457988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.457956    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.457988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.457971    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.457988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.457984    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.458239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.457999    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8234a61-a840-4b87-b7d0-0868567a98e3-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.458239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.458013    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.458239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.458026    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.458239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.458057    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8234a61-a840-4b87-b7d0-0868567a98e3-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.458239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.458072    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8234a61-a840-4b87-b7d0-0868567a98e3-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.805520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.805424    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:52:55.950066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.947280    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4" event={"ID":"a6dedf1a-4f23-403e-b2ce-0d1d1732884a","Type":"ContainerDied","Data":"628e4cf0d7149a071551ef594f03a573c7331a5ed4ad0aac8d58997c20c7c069"}
Oct 31 01:52:55.950066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.947360    5293 scope.go:122] "RemoveContainer" containerID="2cf74b8e731308d41198e136adc525b8a1be943a7436d7bc9ee826ea2544af62"
Oct 31 01:52:55.950066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.947534    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"
Oct 31 01:52:55.989517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.989485    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82" event={"ID":"8eb20b7d-f1b3-4773-be3b-335f1271d3a6","Type":"ContainerDied","Data":"bdd9e929ac023fbbbf63cb9d01788e9ca76959ba576de269ff58adf147206558"}
Oct 31 01:52:55.993919 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:55.989843    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"
Oct 31 01:52:56.053746 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.015223    5293 generic.go:356] "Generic (PLEG): container finished" podID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerID="9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616" exitCode=0
Oct 31 01:52:56.066119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.015249    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" event={"ID":"b8234a61-a840-4b87-b7d0-0868567a98e3","Type":"ContainerDied","Data":"9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616"}
Oct 31 01:52:56.089225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.015366    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"
Oct 31 01:52:56.093226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.089478    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4" event={"ID":"b8234a61-a840-4b87-b7d0-0868567a98e3","Type":"ContainerDied","Data":"73ec13bb3e4a6bc37f11774ee869d3efc356cc93bf79796f68cce6309068443b"}
Oct 31 01:52:56.093401 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.093289    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"]
Oct 31 01:52:56.102478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102446    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy"
Oct 31 01:52:56.102478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102477    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy"
Oct 31 01:52:56.102708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102497    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a6dedf1a-4f23-403e-b2ce-0d1d1732884a" containerName="payload-pre-processing"
Oct 31 01:52:56.102708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102505    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6dedf1a-4f23-403e-b2ce-0d1d1732884a" containerName="payload-pre-processing"
Oct 31 01:52:56.102708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102539    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc" containerName="revoke-keys"
Oct 31 01:52:56.102708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102547    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc" containerName="revoke-keys"
Oct 31 01:52:56.102708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102584    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8eb20b7d-f1b3-4773-be3b-335f1271d3a6" containerName="payload-processing"
Oct 31 01:52:56.102708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102595    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb20b7d-f1b3-4773-be3b-335f1271d3a6" containerName="payload-processing"
Oct 31 01:52:56.102900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102775    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc" containerName="revoke-keys"
Oct 31 01:52:56.102900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102792    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy"
Oct 31 01:52:56.102900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102805    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb20b7d-f1b3-4773-be3b-335f1271d3a6" containerName="payload-processing"
Oct 31 01:52:56.102900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.102822    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6dedf1a-4f23-403e-b2ce-0d1d1732884a" containerName="payload-pre-processing"
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.123184    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"]
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.123210    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"]
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.123762    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy"
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.123772    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy"
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.123875    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" containerName="istio-proxy"
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.125235    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.127288    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40a891-68c7d596d7-frvx4"]
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.127313    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"]
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.127333    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"]
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.130023    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.132249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.131830    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.132887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.132440    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"]
Oct 31 01:52:56.155408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.138947    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:52:56.155408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.139122    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:52:56.155408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.139175    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:52:56.155408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.139186    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"]
Oct 31 01:52:56.155408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.139234    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:52:56.172349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172319    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.172507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172414    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b701fb-9871-4274-a0d0-b11054989cd9-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172442    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.172507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172469    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172519    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172559    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68lk\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-kube-api-access-t68lk\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.172681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172589    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.172681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172649    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172675    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172689    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172721    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172739    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmbs\" (UniqueName: \"kubernetes.io/projected/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-kube-api-access-bfmbs\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172757    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172773    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172794    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172811    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172848    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcq7\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-kube-api-access-6fcq7\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172866    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172886    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.172944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172945    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.173538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.172997    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.174413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.174392    5293 scope.go:122] "RemoveContainer" containerID="11cbcaeebf2435f992f1d773df32b4df964527421291d02652a9809a361d368a"
Oct 31 01:52:56.191148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.188858    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6dedf1a-4f23-403e-b2ce-0d1d1732884a" path="/var/lib/kubelet/pods/a6dedf1a-4f23-403e-b2ce-0d1d1732884a/volumes"
Oct 31 01:52:56.191148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.189358    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d16eaa-b93b-4279-aea4-2add95143e7d" path="/var/lib/kubelet/pods/e9d16eaa-b93b-4279-aea4-2add95143e7d/volumes"
Oct 31 01:52:56.191148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.189686    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"]
Oct 31 01:52:56.191148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.189712    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40a891-674fcbf5bb-7zs82"]
Oct 31 01:52:56.194732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.194713    5293 scope.go:122] "RemoveContainer" containerID="9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616"
Oct 31 01:52:56.206737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.206718    5293 scope.go:122] "RemoveContainer" containerID="2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480"
Oct 31 01:52:56.216002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.215979    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"]
Oct 31 01:52:56.218774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.218755    5293 scope.go:122] "RemoveContainer" containerID="9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616"
Oct 31 01:52:56.219086 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:52:56.219062    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616\": container with ID starting with 9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616 not found: ID does not exist" containerID="9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616"
Oct 31 01:52:56.219155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.219098    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616"} err="failed to get container status \"9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616\": rpc error: code = NotFound desc = could not find container \"9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616\": container with ID starting with 9733e7ce576229626ffec455cb0eba94954ec7956d5a0bce55b7d71b3a63e616 not found: ID does not exist"
Oct 31 01:52:56.219155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.219121    5293 scope.go:122] "RemoveContainer" containerID="2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480"
Oct 31 01:52:56.219387 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:52:56.219368    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480\": container with ID starting with 2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480 not found: ID does not exist" containerID="2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480"
Oct 31 01:52:56.219448 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.219395    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480"} err="failed to get container status \"2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480\": rpc error: code = NotFound desc = could not find container \"2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480\": container with ID starting with 2e6577dd87bf4cab9047890d23a5616bab164e90529f95778e14a3433f515480 not found: ID does not exist"
Oct 31 01:52:56.226510 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.226489    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-v48p4"]
Oct 31 01:52:56.274630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274605    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.274801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274645    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.274801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274772    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.274941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274842    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.274941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274880    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmbs\" (UniqueName: \"kubernetes.io/projected/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-kube-api-access-bfmbs\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.274941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274916    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.275149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274943    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274984    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.274990    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275117    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275184    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcq7\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-kube-api-access-6fcq7\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275215    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275233    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.275392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275252    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275294    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275361    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275440    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275495    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275605    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b701fb-9871-4274-a0d0-b11054989cd9-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275634    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275663    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275695    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275708    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275803    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275865    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68lk\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-kube-api-access-t68lk\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.275960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275258    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.275960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.275921    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.277106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.276275    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.277106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.276387    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b701fb-9871-4274-a0d0-b11054989cd9-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.277106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.276476    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.277106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.276588    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.277106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.276837    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.278666 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.278633    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.278666 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.278652    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.278810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.278684    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.279398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.279378    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.279501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.279421    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.287868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.287845    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.288006 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.287989    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68lk\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-kube-api-access-t68lk\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.288240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.288216    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.290620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.290600    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmbs\" (UniqueName: \"kubernetes.io/projected/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-kube-api-access-bfmbs\") pod \"maas-api-revoke-sub-a3e7b37a-rgsbf\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.290948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.290926    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcq7\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-kube-api-access-6fcq7\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.461955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.461921    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:52:56.491995 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.491961    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:56.495150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.495121    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:56.754302 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:52:56.754265    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fcdfcb_af8d_4dc5_80e6_46d7e8b05a06.slice/crio-e2bcd807e9d5779c35f5c18a5de077f5aa5fb1362df3ccedd05fb8663ab66556 WatchSource:0}: Error finding container e2bcd807e9d5779c35f5c18a5de077f5aa5fb1362df3ccedd05fb8663ab66556: Status 404 returned error can't find the container with id e2bcd807e9d5779c35f5c18a5de077f5aa5fb1362df3ccedd05fb8663ab66556
Oct 31 01:52:56.754564 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.754537    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"]
Oct 31 01:52:56.835363 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.835331    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"]
Oct 31 01:52:56.836146 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:52:56.836120    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb546a5_2b16_4018_a783_07a8fd2bacf7.slice/crio-4a7ed61d460d94091b3c65e5c2ff0926fc2c8960097bbbd8c99c69d835ec131d WatchSource:0}: Error finding container 4a7ed61d460d94091b3c65e5c2ff0926fc2c8960097bbbd8c99c69d835ec131d: Status 404 returned error can't find the container with id 4a7ed61d460d94091b3c65e5c2ff0926fc2c8960097bbbd8c99c69d835ec131d
Oct 31 01:52:56.839410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.839367    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:56.839523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.839475    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:56.839584 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.839522    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:56.872066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.871366    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"]
Oct 31 01:52:56.883395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.880597    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:56.883395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.880715    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:56.883395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:56.880772    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:52:57.420588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:57.420530    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" event={"ID":"b1b701fb-9871-4274-a0d0-b11054989cd9","Type":"ContainerStarted","Data":"c3298a83365f8bec18082f2ad4796193897939da6a657c62de0b4033e8497348"}
Oct 31 01:52:57.462008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:57.461965    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf" event={"ID":"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06","Type":"ContainerStarted","Data":"e2bcd807e9d5779c35f5c18a5de077f5aa5fb1362df3ccedd05fb8663ab66556"}
Oct 31 01:52:57.468750 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:57.468712    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" event={"ID":"1eb546a5-2b16-4018-a783-07a8fd2bacf7","Type":"ContainerStarted","Data":"4a7ed61d460d94091b3c65e5c2ff0926fc2c8960097bbbd8c99c69d835ec131d"}
Oct 31 01:52:58.235072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:58.231146    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb20b7d-f1b3-4773-be3b-335f1271d3a6" path="/var/lib/kubelet/pods/8eb20b7d-f1b3-4773-be3b-335f1271d3a6/volumes"
Oct 31 01:52:58.235072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:58.232068    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8234a61-a840-4b87-b7d0-0868567a98e3" path="/var/lib/kubelet/pods/b8234a61-a840-4b87-b7d0-0868567a98e3/volumes"
Oct 31 01:52:58.506064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:58.505967    5293 generic.go:356] "Generic (PLEG): container finished" podID="f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06" containerID="e2de11a6d059064c64a8bca80a0e24fbcf4194822ebe65438757ea236d371383" exitCode=0
Oct 31 01:52:58.506577 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:58.506120    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf" event={"ID":"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06","Type":"ContainerDied","Data":"e2de11a6d059064c64a8bca80a0e24fbcf4194822ebe65438757ea236d371383"}
Oct 31 01:52:58.568069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:58.565205    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" event={"ID":"1eb546a5-2b16-4018-a783-07a8fd2bacf7","Type":"ContainerStarted","Data":"5f07e899d95f8d4dbe4a2905433aff3fd337a83d78bc3f2d1d8de254e67f9f47"}
Oct 31 01:52:58.578824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:58.578786    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" event={"ID":"b1b701fb-9871-4274-a0d0-b11054989cd9","Type":"ContainerStarted","Data":"b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059"}
Oct 31 01:52:58.617341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:58.617274    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podStartSLOduration=3.617253025 podStartE2EDuration="3.617253025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:58.590548331 +0000 UTC m=+1936.539196736" watchObservedRunningTime="2026-10-31 01:52:58.617253025 +0000 UTC m=+1936.565901418"
Oct 31 01:52:58.719111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:58.694985    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podStartSLOduration=2.694962002 podStartE2EDuration="2.694962002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:58.622724371 +0000 UTC m=+1936.571372761" watchObservedRunningTime="2026-10-31 01:52:58.694962002 +0000 UTC m=+1936.643610396"
Oct 31 01:52:59.164870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.140267    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:52:59.420239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.420117    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:52:59.466895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.466758    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:52:59.471907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.470314    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:52:59.552068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.500445    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:52:59.557232 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.557199    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:52:59.561062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.512479    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:52:59.561316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.561285    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:52:59.565062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.514286    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:52:59.565300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.565271    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:52:59.612478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.612427    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:52:59.618076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.617961    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:52:59.710432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.710334    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:52:59.871358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:52:59.871316    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:00.026875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.026135    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:00.057212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.057156    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:00.113330 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.093471    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:53:00.144264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.142447    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:53:00.146094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.144945    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:53:00.544189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.478700    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:00.544189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.518908    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:00.544189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.518962    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:00.569096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.548463    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:53:00.569096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.549957    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:53:00.569096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.550001    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:53:00.574067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.571615    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:53:00.627177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.614337    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:53:00.628777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.628747    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:53:00.691167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.639846    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:53:00.691167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.639920    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:53:00.691167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.639958    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:53:00.972069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:00.936587    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:01.074063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.050291    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:01.074063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.065248    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:01.102067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.076694    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:53:01.102067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.076793    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:53:01.151163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.151121    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:01.175067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.152737    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:53:01.175067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.170003    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:53:01.212171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.204492    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:53:01.319655 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.319562    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf" event={"ID":"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06","Type":"ContainerDied","Data":"e2bcd807e9d5779c35f5c18a5de077f5aa5fb1362df3ccedd05fb8663ab66556"}
Oct 31 01:53:01.319951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.319927    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2bcd807e9d5779c35f5c18a5de077f5aa5fb1362df3ccedd05fb8663ab66556"
Oct 31 01:53:01.457474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.394869    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:01.519588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.519552    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:01.519886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.519846    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:01.579113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.554607    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:53:01.620070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.618222    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.644177    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"]
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.645777    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06" containerName="revoke-keys"
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.645799    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06" containerName="revoke-keys"
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.646250    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06" containerName="revoke-keys"
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.654822    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-openshift-ca\") pod \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") "
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.655021    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-kube-api-access-bfmbs\" (UniqueName: \"kubernetes.io/projected/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-kube-api-access-bfmbs\") pod \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") "
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.655172    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-xks-ca\") pod \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\" (UID: \"f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06\") "
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.674444    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.675350    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:01.678160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.677466    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:01.686595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.686567    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-openshift-ca" pod "f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06" (UID: "f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:01.906686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.906619    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:53:01.958645 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.958611    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"]
Oct 31 01:53:01.959167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.959146    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:53:01.959454 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.959441    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:53:01.959662 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.959651    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:53:01.960018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:01.959984    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.118897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.067329    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:02.118897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.110795    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.118897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.110909    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.118897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.111130    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxf7\" (UniqueName: \"kubernetes.io/projected/be6698e9-d63b-4aa4-9210-50d9560fa3c6-kube-api-access-bsxf7\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.128457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.128432    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:02.331529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.320645    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.331529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.320774    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.331529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.320975    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxf7\" (UniqueName: \"kubernetes.io/projected/be6698e9-d63b-4aa4-9210-50d9560fa3c6-kube-api-access-bsxf7\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.331529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.323140    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-kube-api-access-bfmbs" pod "f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06" (UID: "f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06"). InnerVolumeSpecName "kube-api-access-bfmbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:02.394249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.393652    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-xks-ca" pod "f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06" (UID: "f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:02.434063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.419638    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:02.434063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.419684    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmbs\" (UniqueName: \"kubernetes.io/projected/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06-kube-api-access-bfmbs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:02.692217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.692164    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:02.790960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.693158    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:02.824872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.815329    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:02.824872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.815393    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:02.824872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.707333    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.824872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.707188    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:02.824872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.823214    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:53:02.868076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.865970    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:02.868076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.866173    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:53:02.868076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.866217    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:53:02.868076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.866644    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:02.944144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.913576    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:02.944144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.923613    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:53:02.945884 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.945182    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:02.992063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:02.965804    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"
Oct 31 01:53:03.036066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.002754    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxf7\" (UniqueName: \"kubernetes.io/projected/be6698e9-d63b-4aa4-9210-50d9560fa3c6-kube-api-access-bsxf7\") pod \"maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:03.036066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.009141    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:03.055480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.055440    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 31 01:53:03.057017 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.056974    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:03.108368 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.108326    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:53:03.205126 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.181395    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:03.354472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.354435    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"]
Oct 31 01:53:03.433308 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.397675    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy" containerID="cri-o://eea4468a68755965fbf11171de155152fa220d42a33cf7eb8516ed325d503a78" gracePeriod=30
Oct 31 01:53:03.433308 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.399259    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:03.492519 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.492489    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:03.492812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.492771    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:03.495901 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.495877    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:53:03.496084 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.496026    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:53:03.792806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.791148    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:53:03.792806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.791844    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:03.792806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.792221    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:53:03.858251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.856152    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:03.858251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.856172    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:53:03.858251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.857117    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:53:03.906065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.888655    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 31 01:53:03.906065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.888869    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:53:03.906065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.889388    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:53:03.967088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.942883    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:53:03.984349 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:03.958270    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:04.012924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.009753    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:53:04.012924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.010332    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:53:04.059232 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.059130    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:04.059847 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.059822    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:53:04.145163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.145123    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:53:04.159060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.145511    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:53:04.262118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.201356    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:04.563116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.542555    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:53:04.563116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.542610    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:53:04.563116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.543780    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:04.563116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.543825    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:04.686562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.686514    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:04.759094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.727810    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:53:04.800761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.800733    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:53:04.802381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.796090    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:04.811132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.797912    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:53:04.929952 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.927736    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:04.942203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:04.938183    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:05.068065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.006545    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:05.119322 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.094717    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:05.119322 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.101133    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:05.160205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.160155    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:05.370470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.358195    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:53:05.395397 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.378979    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:05.502140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.494709    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:05.502140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.494775    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:05.502140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.495563    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"]
Oct 31 01:53:05.528080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.516096    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:53:05.528080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.516155    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:53:05.592835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.592802    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:53:05.759574 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.759465    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-278a56-66gjd"]
Oct 31 01:53:05.791069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.781031    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:53:05.791069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.781154    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:53:05.833069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.806007    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:05.834614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.833646    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:05.967386 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:05.966652    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:53:06.010995 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.010895    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:53:06.119413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.012898    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"]
Oct 31 01:53:06.119719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.098475    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:06.323081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.322173    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:06.323081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.322505    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:53:06.554203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.546315    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:06.554203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.546378    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:06.554203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.546533    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:53:06.554203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.546566    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:53:06.780790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.780746    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:06.836208 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.836112    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:53:06.856189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.856146    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:06.995845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.995799    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc" path="/var/lib/kubelet/pods/b49e4a26-0b38-4fd9-8b1d-bbd818f2c2dc/volumes"
Oct 31 01:53:07.005070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:06.996308    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:53:07.005305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.005279    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc" event={"ID":"be6698e9-d63b-4aa4-9210-50d9560fa3c6","Type":"ContainerStarted","Data":"05f0958ac5a2fe92e741e9d2c3b880180f6b2d7a2c012a6a5b90ad3e8bcf321b"}
Oct 31 01:53:07.019219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.019175    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:53:07.020245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.020215    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:53:07.031115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.022297    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:53:07.085656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.084880    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:53:07.087102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.087024    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:53:07.090777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.090718    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:53:07.090976 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.090962    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:53:07.091191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.090718    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:07.113237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.113208    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:53:07.134470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.125814    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:53:07.287726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.284316    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:53:07.492396 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.492363    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:07.492601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.492415    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:07.495429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.495403    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 31 01:53:07.495539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.495443    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 31 01:53:07.843098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.842965    5293 generic.go:356] "Generic (PLEG): container finished" podID="be6698e9-d63b-4aa4-9210-50d9560fa3c6" containerID="a68baec8a0345923487e92ffb9bba43145ea7a12c590d06627ef0ad8520a7638" exitCode=0
Oct 31 01:53:07.843589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:07.843402    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc" event={"ID":"be6698e9-d63b-4aa4-9210-50d9560fa3c6","Type":"ContainerDied","Data":"a68baec8a0345923487e92ffb9bba43145ea7a12c590d06627ef0ad8520a7638"}
Oct 31 01:53:08.492793 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:08.492758    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:53:08.492979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:08.492820    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 31 01:53:08.872116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:08.871341    5293 generic.go:356] "Generic (PLEG): container finished" podID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerID="eea4468a68755965fbf11171de155152fa220d42a33cf7eb8516ed325d503a78" exitCode=0
Oct 31 01:53:08.872116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:08.871814    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" event={"ID":"e1942154-283c-4d10-acb4-4a05d07a8ac5","Type":"ContainerDied","Data":"eea4468a68755965fbf11171de155152fa220d42a33cf7eb8516ed325d503a78"}
Oct 31 01:53:08.872116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:08.871875    5293 scope.go:122] "RemoveContainer" containerID="cbb4796c756c3ad3ad41b6a2c3a4cb12604810c479c2deed9e5cf88f141c3edc"
Oct 31 01:53:09.220298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.219736    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:53:09.220576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.220523    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:09.243764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.242280    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:09.322656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.322612    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-xks-ca\") pod \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") "
Oct 31 01:53:09.322850 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.322784    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6698e9-d63b-4aa4-9210-50d9560fa3c6-kube-api-access-bsxf7\" (UniqueName: \"kubernetes.io/projected/be6698e9-d63b-4aa4-9210-50d9560fa3c6-kube-api-access-bsxf7\") pod \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") "
Oct 31 01:53:09.322933 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.322859    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-openshift-ca\") pod \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\" (UID: \"be6698e9-d63b-4aa4-9210-50d9560fa3c6\") "
Oct 31 01:53:09.349078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.347297    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-openshift-ca" pod "be6698e9-d63b-4aa4-9210-50d9560fa3c6" (UID: "be6698e9-d63b-4aa4-9210-50d9560fa3c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:09.349078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.347882    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:53:09.349078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.348855    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:53:09.351974 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.349980    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 01:53:09.354214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.354159    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6698e9-d63b-4aa4-9210-50d9560fa3c6-kube-api-access-bsxf7" pod "be6698e9-d63b-4aa4-9210-50d9560fa3c6" (UID: "be6698e9-d63b-4aa4-9210-50d9560fa3c6"). InnerVolumeSpecName "kube-api-access-bsxf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:09.355014 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.354990    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-xks-ca" pod "be6698e9-d63b-4aa4-9210-50d9560fa3c6" (UID: "be6698e9-d63b-4aa4-9210-50d9560fa3c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:09.439529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.439487    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.440375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.440349    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxf7\" (UniqueName: \"kubernetes.io/projected/be6698e9-d63b-4aa4-9210-50d9560fa3c6-kube-api-access-bsxf7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.440561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.440546    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6698e9-d63b-4aa4-9210-50d9560fa3c6-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.554992 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.554965    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654243    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-envoy\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654345    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-socket\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654458    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-certs\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654490    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-kube-api-access-vrftf\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-kube-api-access-vrftf\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654590    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-token\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-token\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654628    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-podinfo\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654670    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-credential-socket\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654771    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1942154-283c-4d10-acb4-4a05d07a8ac5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1942154-283c-4d10-acb4-4a05d07a8ac5-istiod-ca-cert\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.657141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.654862    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-data\") pod \"e1942154-283c-4d10-acb4-4a05d07a8ac5\" (UID: \"e1942154-283c-4d10-acb4-4a05d07a8ac5\") "
Oct 31 01:53:09.680713 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.680638    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-socket" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:09.681021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.680891    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-certs" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:09.687077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.686950    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-envoy" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:09.687653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.687448    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-credential-socket" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:09.687952 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.687933    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1942154-283c-4d10-acb4-4a05d07a8ac5-istiod-ca-cert" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:09.709026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.708983    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-data" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:09.734593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.734556    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-kube-api-access-vrftf" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "kube-api-access-vrftf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:09.740421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.736105    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-podinfo" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:53:09.745078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.743106    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"]
Oct 31 01:53:09.746729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.746660    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy"
Oct 31 01:53:09.746729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.746680    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy"
Oct 31 01:53:09.746729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.746721    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy"
Oct 31 01:53:09.746729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.746731    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy"
Oct 31 01:53:09.747124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.746783    5293 cpu_manager.go:414] "containerMap: removing container" podUID="be6698e9-d63b-4aa4-9210-50d9560fa3c6" containerName="revoke-keys"
Oct 31 01:53:09.747124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.746794    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6698e9-d63b-4aa4-9210-50d9560fa3c6" containerName="revoke-keys"
Oct 31 01:53:09.750404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.749895    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy"
Oct 31 01:53:09.750404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.749941    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" containerName="istio-proxy"
Oct 31 01:53:09.750404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.749966    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6698e9-d63b-4aa4-9210-50d9560fa3c6" containerName="revoke-keys"
Oct 31 01:53:09.765115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.765076    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-token" pod "e1942154-283c-4d10-acb4-4a05d07a8ac5" (UID: "e1942154-283c-4d10-acb4-4a05d07a8ac5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:09.771142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.771115    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:53:09.773160 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.773137    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:09.775374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.775023    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:53:09.778093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.777697    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.778283 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.778260    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 01:53:09.778543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.778527    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.778650 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.778637    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.778741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.778730    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1942154-283c-4d10-acb4-4a05d07a8ac5-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.778836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.778825    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.778920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.778910    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.779004 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.778993    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.779108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.779097    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1942154-283c-4d10-acb4-4a05d07a8ac5-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.779179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.779169    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrftf\" (UniqueName: \"kubernetes.io/projected/e1942154-283c-4d10-acb4-4a05d07a8ac5-kube-api-access-vrftf\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:09.789638 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.789614    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"]
Oct 31 01:53:09.884121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.884082    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5" event={"ID":"e1942154-283c-4d10-acb4-4a05d07a8ac5","Type":"ContainerDied","Data":"5718ff5c7c27505911e556fac20495915fe49821cc44bfa3fc9325d312a08675"}
Oct 31 01:53:09.884495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.884150    5293 scope.go:122] "RemoveContainer" containerID="eea4468a68755965fbf11171de155152fa220d42a33cf7eb8516ed325d503a78"
Oct 31 01:53:09.884495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.884298    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"
Oct 31 01:53:09.885495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.885177    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:09.885495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.885280    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6kd\" (UniqueName: \"kubernetes.io/projected/3084e146-3fa1-4cca-9221-2d02f59bf2e5-kube-api-access-6m6kd\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:09.885686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.885588    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:09.887915 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.887886    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc" event={"ID":"be6698e9-d63b-4aa4-9210-50d9560fa3c6","Type":"ContainerDied","Data":"05f0958ac5a2fe92e741e9d2c3b880180f6b2d7a2c012a6a5b90ad3e8bcf321b"}
Oct 31 01:53:09.888026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.887939    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f0958ac5a2fe92e741e9d2c3b880180f6b2d7a2c012a6a5b90ad3e8bcf321b"
Oct 31 01:53:09.888026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.888005    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"
Oct 31 01:53:09.924285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.924253    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"]
Oct 31 01:53:09.929726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.929696    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-40a891-openshift-default-74488b79bc-6xff5"]
Oct 31 01:53:09.987083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.987027    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:09.987296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.987143    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6kd\" (UniqueName: \"kubernetes.io/projected/3084e146-3fa1-4cca-9221-2d02f59bf2e5-kube-api-access-6m6kd\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:09.987682 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.987466    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:09.988139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.988117    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:09.989930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:09.989913    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:10.014796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.014717    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6kd\" (UniqueName: \"kubernetes.io/projected/3084e146-3fa1-4cca-9221-2d02f59bf2e5-kube-api-access-6m6kd\") pod \"maas-api-revoke-sub-9dc6cc10-6l6dj\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:10.108970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.108712    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:10.181621 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.181497    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1942154-283c-4d10-acb4-4a05d07a8ac5" path="/var/lib/kubelet/pods/e1942154-283c-4d10-acb4-4a05d07a8ac5/volumes"
Oct 31 01:53:10.376369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.376287    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"]
Oct 31 01:53:10.381999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.381974    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:10.388340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.388288    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5f65e1-dockercfg-mxj9p\""
Oct 31 01:53:10.388493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.388443    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5f65e1\""
Oct 31 01:53:10.393840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.393130    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"]
Oct 31 01:53:10.417787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.417757    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"]
Oct 31 01:53:10.431970 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:10.431924    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3084e146_3fa1_4cca_9221_2d02f59bf2e5.slice/crio-a87980e8275ad9a6fffacf3ac21c6657b32462da59f2ebf5a45da4ba5baf67c1 WatchSource:0}: Error finding container a87980e8275ad9a6fffacf3ac21c6657b32462da59f2ebf5a45da4ba5baf67c1: Status 404 returned error can't find the container with id a87980e8275ad9a6fffacf3ac21c6657b32462da59f2ebf5a45da4ba5baf67c1
Oct 31 01:53:10.433359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.433336    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:10.462699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.452157    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"]
Oct 31 01:53:10.540677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.540630    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"]
Oct 31 01:53:10.540990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.520226    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrtv\" (UniqueName: \"kubernetes.io/projected/8f8ccd6c-d000-4828-8d9a-7d8765111534-kube-api-access-tjrtv\") pod \"payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf\" (UID: \"8f8ccd6c-d000-4828-8d9a-7d8765111534\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:10.541594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.541555    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44079c0-9086-441e-8711-1b85236abb2b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk\" (UID: \"d44079c0-9086-441e-8711-1b85236abb2b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:10.541900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.541882    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8ccd6c-d000-4828-8d9a-7d8765111534-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf\" (UID: \"8f8ccd6c-d000-4828-8d9a-7d8765111534\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:10.542282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.542262    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gp26\" (UniqueName: \"kubernetes.io/projected/d44079c0-9086-441e-8711-1b85236abb2b-kube-api-access-7gp26\") pod \"payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk\" (UID: \"d44079c0-9086-441e-8711-1b85236abb2b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:10.712181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.712143    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrtv\" (UniqueName: \"kubernetes.io/projected/8f8ccd6c-d000-4828-8d9a-7d8765111534-kube-api-access-tjrtv\") pod \"payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf\" (UID: \"8f8ccd6c-d000-4828-8d9a-7d8765111534\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:10.712451 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.712427    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44079c0-9086-441e-8711-1b85236abb2b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk\" (UID: \"d44079c0-9086-441e-8711-1b85236abb2b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:10.712633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.712619    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8ccd6c-d000-4828-8d9a-7d8765111534-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf\" (UID: \"8f8ccd6c-d000-4828-8d9a-7d8765111534\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:10.712827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.712805    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gp26\" (UniqueName: \"kubernetes.io/projected/d44079c0-9086-441e-8711-1b85236abb2b-kube-api-access-7gp26\") pod \"payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk\" (UID: \"d44079c0-9086-441e-8711-1b85236abb2b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:10.714525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.714348    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44079c0-9086-441e-8711-1b85236abb2b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk\" (UID: \"d44079c0-9086-441e-8711-1b85236abb2b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:10.715780 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.715747    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8ccd6c-d000-4828-8d9a-7d8765111534-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf\" (UID: \"8f8ccd6c-d000-4828-8d9a-7d8765111534\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:10.746257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.740670    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrtv\" (UniqueName: \"kubernetes.io/projected/8f8ccd6c-d000-4828-8d9a-7d8765111534-kube-api-access-tjrtv\") pod \"payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf\" (UID: \"8f8ccd6c-d000-4828-8d9a-7d8765111534\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:10.750666 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.750611    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gp26\" (UniqueName: \"kubernetes.io/projected/d44079c0-9086-441e-8711-1b85236abb2b-kube-api-access-7gp26\") pod \"payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk\" (UID: \"d44079c0-9086-441e-8711-1b85236abb2b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:10.974077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.962738    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:10.984354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:10.984276    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj" event={"ID":"3084e146-3fa1-4cca-9221-2d02f59bf2e5","Type":"ContainerStarted","Data":"a87980e8275ad9a6fffacf3ac21c6657b32462da59f2ebf5a45da4ba5baf67c1"}
Oct 31 01:53:11.013907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.012577    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:11.111667 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.105286    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"]
Oct 31 01:53:11.314492 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.314465    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"]
Oct 31 01:53:11.316343 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:11.316316    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8ccd6c_d000_4828_8d9a_7d8765111534.slice/crio-b93718fb50c9ee9249af2afbdaa131f17c0569ea484c72bd40eb6896c27700c1 WatchSource:0}: Error finding container b93718fb50c9ee9249af2afbdaa131f17c0569ea484c72bd40eb6896c27700c1: Status 404 returned error can't find the container with id b93718fb50c9ee9249af2afbdaa131f17c0569ea484c72bd40eb6896c27700c1
Oct 31 01:53:11.387798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.387774    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"]
Oct 31 01:53:11.389287 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:11.389261    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44079c0_9086_441e_8711_1b85236abb2b.slice/crio-0c27d4c097faee50ad7e3016891c3fdb437eaed5b302e1dfbe4e0a4d4610ed3e WatchSource:0}: Error finding container 0c27d4c097faee50ad7e3016891c3fdb437eaed5b302e1dfbe4e0a4d4610ed3e: Status 404 returned error can't find the container with id 0c27d4c097faee50ad7e3016891c3fdb437eaed5b302e1dfbe4e0a4d4610ed3e
Oct 31 01:53:11.779356 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:53:11.779300    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:53:11.993514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.993483    5293 generic.go:356] "Generic (PLEG): container finished" podID="3084e146-3fa1-4cca-9221-2d02f59bf2e5" containerID="0ab0243639be49a493b84a0a34c4afd294281d22382e3b02853198fd60eed3a2" exitCode=0
Oct 31 01:53:11.994011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.993584    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj" event={"ID":"3084e146-3fa1-4cca-9221-2d02f59bf2e5","Type":"ContainerDied","Data":"0ab0243639be49a493b84a0a34c4afd294281d22382e3b02853198fd60eed3a2"}
Oct 31 01:53:11.995774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.995751    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf" event={"ID":"8f8ccd6c-d000-4828-8d9a-7d8765111534","Type":"ContainerStarted","Data":"a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d"}
Oct 31 01:53:11.995978 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.995961    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:11.996175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.996160    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf" event={"ID":"8f8ccd6c-d000-4828-8d9a-7d8765111534","Type":"ContainerStarted","Data":"b93718fb50c9ee9249af2afbdaa131f17c0569ea484c72bd40eb6896c27700c1"}
Oct 31 01:53:11.998639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.998545    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk" event={"ID":"d44079c0-9086-441e-8711-1b85236abb2b","Type":"ContainerStarted","Data":"6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e"}
Oct 31 01:53:11.998639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.998579    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk" event={"ID":"d44079c0-9086-441e-8711-1b85236abb2b","Type":"ContainerStarted","Data":"0c27d4c097faee50ad7e3016891c3fdb437eaed5b302e1dfbe4e0a4d4610ed3e"}
Oct 31 01:53:11.998639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:11.998616    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:12.030866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:12.030806    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf" podStartSLOduration=2.030788396 podStartE2EDuration="2.030788396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:12.029411394 +0000 UTC m=+1949.978059787" watchObservedRunningTime="2026-10-31 01:53:12.030788396 +0000 UTC m=+1949.979436790"
Oct 31 01:53:12.053088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:12.053018    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk" podStartSLOduration=2.053000362 podStartE2EDuration="2.053000362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:12.052199453 +0000 UTC m=+1950.000847845" watchObservedRunningTime="2026-10-31 01:53:12.053000362 +0000 UTC m=+1950.001648756"
Oct 31 01:53:13.205188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.205115    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"]
Oct 31 01:53:13.206096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.206062    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy" containerID="cri-o://8a77c40754a1f835eb94a54a4b38f7c7eb39d9bf4f087f3701c4e8c957c15138" gracePeriod=30
Oct 31 01:53:13.263096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.263073    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:13.267206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.267181    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3084e146-3fa1-4cca-9221-2d02f59bf2e5-kube-api-access-6m6kd\" (UniqueName: \"kubernetes.io/projected/3084e146-3fa1-4cca-9221-2d02f59bf2e5-kube-api-access-6m6kd\") pod \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") "
Oct 31 01:53:13.267447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.267425    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-xks-ca\") pod \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") "
Oct 31 01:53:13.267607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.267587    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-openshift-ca\") pod \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\" (UID: \"3084e146-3fa1-4cca-9221-2d02f59bf2e5\") "
Oct 31 01:53:13.267965 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.267937    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-openshift-ca" pod "3084e146-3fa1-4cca-9221-2d02f59bf2e5" (UID: "3084e146-3fa1-4cca-9221-2d02f59bf2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:13.285398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.269419    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:13.285398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.269827    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-xks-ca" pod "3084e146-3fa1-4cca-9221-2d02f59bf2e5" (UID: "3084e146-3fa1-4cca-9221-2d02f59bf2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:13.285398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.270031    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3084e146-3fa1-4cca-9221-2d02f59bf2e5-kube-api-access-6m6kd" pod "3084e146-3fa1-4cca-9221-2d02f59bf2e5" (UID: "3084e146-3fa1-4cca-9221-2d02f59bf2e5"). InnerVolumeSpecName "kube-api-access-6m6kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:13.375122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.373384    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6kd\" (UniqueName: \"kubernetes.io/projected/3084e146-3fa1-4cca-9221-2d02f59bf2e5-kube-api-access-6m6kd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:13.375122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:13.373421    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3084e146-3fa1-4cca-9221-2d02f59bf2e5-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:14.084995 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.084953    5293 generic.go:356] "Generic (PLEG): container finished" podID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerID="9a2f37bc55d219e810735f18858dd17a05c378eca95482c3b29686ec9f48cd81" exitCode=0
Oct 31 01:53:14.085473 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.085443    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" event={"ID":"a1c3959c-9fd5-4a05-b004-a27bab56e7a5","Type":"ContainerDied","Data":"9a2f37bc55d219e810735f18858dd17a05c378eca95482c3b29686ec9f48cd81"}
Oct 31 01:53:14.086229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.086206    5293 scope.go:122] "RemoveContainer" containerID="9a2f37bc55d219e810735f18858dd17a05c378eca95482c3b29686ec9f48cd81"
Oct 31 01:53:14.088889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.088837    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:14.169761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.165244    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:14.173074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.167931    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:14.264063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.259988    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"
Oct 31 01:53:14.283627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.283587    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj" event={"ID":"3084e146-3fa1-4cca-9221-2d02f59bf2e5","Type":"ContainerDied","Data":"a87980e8275ad9a6fffacf3ac21c6657b32462da59f2ebf5a45da4ba5baf67c1"}
Oct 31 01:53:14.283847 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.283828    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87980e8275ad9a6fffacf3ac21c6657b32462da59f2ebf5a45da4ba5baf67c1"
Oct 31 01:53:14.888062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.887983    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"]
Oct 31 01:53:14.895129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.892983    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3084e146-3fa1-4cca-9221-2d02f59bf2e5" containerName="revoke-keys"
Oct 31 01:53:14.895129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.893015    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3084e146-3fa1-4cca-9221-2d02f59bf2e5" containerName="revoke-keys"
Oct 31 01:53:14.895129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:14.893271    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3084e146-3fa1-4cca-9221-2d02f59bf2e5" containerName="revoke-keys"
Oct 31 01:53:15.012526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.012487    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"]
Oct 31 01:53:15.012876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.012855    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.030257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.030225    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:15.057267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.057238    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:15.057580 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.057526    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:15.126750 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.126370    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:53:15.220068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.217683    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.220068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.217773    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg5wl\" (UniqueName: \"kubernetes.io/projected/71a4f410-7615-498e-bfc3-38297690ae6a-kube-api-access-jg5wl\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.220068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.217878    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.407066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.404650    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.407066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.404742    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg5wl\" (UniqueName: \"kubernetes.io/projected/71a4f410-7615-498e-bfc3-38297690ae6a-kube-api-access-jg5wl\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.407066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.404840    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.407066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.405729    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.505549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.505418    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.562099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.561588    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg5wl\" (UniqueName: \"kubernetes.io/projected/71a4f410-7615-498e-bfc3-38297690ae6a-kube-api-access-jg5wl\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.583085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.583024    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" event={"ID":"a1c3959c-9fd5-4a05-b004-a27bab56e7a5","Type":"ContainerStarted","Data":"8b13a438d6e1627aee08b44e671e0c22b184652f0d9037b53759ba1e84de53ca"}
Oct 31 01:53:15.793257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.793150    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:15.808278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.808242    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:15.808840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.808807    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:15.862963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.862920    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:15.978630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:15.970218    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:53:16.041235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.041187    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:16.114120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.110937    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk" podUID="2f6490bd-3a92-467b-9c0e-346efcc86a5a" containerName="payload-processing" containerID="cri-o://8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883" gracePeriod=25
Oct 31 01:53:16.114120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.111309    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:16.295534 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.295486    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:16.513142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.513108    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:53:16.513769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.513747    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 01:53:16.513959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.513935    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 01:53:16.600340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.600297    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:16.859554 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.859465    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"]
Oct 31 01:53:16.873283 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.873248    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"]
Oct 31 01:53:16.873475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.873351    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:16.880030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.879986    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk" podUID="2f6490bd-3a92-467b-9c0e-346efcc86a5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:53:16.940465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:16.940356    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"]
Oct 31 01:53:17.036155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.036121    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqgg5\" (UniqueName: \"kubernetes.io/projected/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-kube-api-access-wqgg5\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.036342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.036323    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.036536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.036521    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.138421 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.138392    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqgg5\" (UniqueName: \"kubernetes.io/projected/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-kube-api-access-wqgg5\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.138584 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.138515    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.138677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.138659    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.139183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.139161    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.141360 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.141336    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.152123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.152097    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqgg5\" (UniqueName: \"kubernetes.io/projected/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-kube-api-access-wqgg5\") pod \"maas-api-revoke-sub-37ea0a58-t4skz\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.195566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.195274    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:17.209966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.209872    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:53:17.209966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.209932    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:53:17.524656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.524220    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 01:53:17.524656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.524279    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 01:53:17.629369 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:17.629338    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06b6bc6_0460_4d28_85d5_eacbbe50a06c.slice/crio-0f42e87c9855b653152922dc1cd9186adb788756517db363e9efdf498de4f870 WatchSource:0}: Error finding container 0f42e87c9855b653152922dc1cd9186adb788756517db363e9efdf498de4f870: Status 404 returned error can't find the container with id 0f42e87c9855b653152922dc1cd9186adb788756517db363e9efdf498de4f870
Oct 31 01:53:17.629770 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.629742    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"]
Oct 31 01:53:17.660359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.660329    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz" event={"ID":"d06b6bc6-0460-4d28-85d5-eacbbe50a06c","Type":"ContainerStarted","Data":"0f42e87c9855b653152922dc1cd9186adb788756517db363e9efdf498de4f870"}
Oct 31 01:53:17.661886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.661853    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x" event={"ID":"71a4f410-7615-498e-bfc3-38297690ae6a","Type":"ContainerStarted","Data":"6ad7f49b6ece25cc19e5e7ec6e923d2f6c9a67c5d9c7063100a3df145b7da6a9"}
Oct 31 01:53:17.661990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.661901    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x" event={"ID":"71a4f410-7615-498e-bfc3-38297690ae6a","Type":"ContainerStarted","Data":"b36062cf266446d5ac584f64e103e375c3dbbf55f36b941a0a1e58c14db3118f"}
Oct 31 01:53:17.680722 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:17.680672    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x" podStartSLOduration=3.680658976 podStartE2EDuration="3.680658976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:17.678437083 +0000 UTC m=+1955.627085475" watchObservedRunningTime="2026-10-31 01:53:17.680658976 +0000 UTC m=+1955.629307368"
Oct 31 01:53:18.494810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.494780    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 01:53:18.495001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.494832    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 01:53:18.675953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.675905    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz" event={"ID":"d06b6bc6-0460-4d28-85d5-eacbbe50a06c","Type":"ContainerStarted","Data":"feca8957f1d7d5e51f0d33a7b767326f96be8c7cefc761bbb140a688a1453801"}
Oct 31 01:53:18.678796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.678767    5293 generic.go:356] "Generic (PLEG): container finished" podID="71a4f410-7615-498e-bfc3-38297690ae6a" containerID="6ad7f49b6ece25cc19e5e7ec6e923d2f6c9a67c5d9c7063100a3df145b7da6a9" exitCode=0
Oct 31 01:53:18.679087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.678845    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x" event={"ID":"71a4f410-7615-498e-bfc3-38297690ae6a","Type":"ContainerDied","Data":"6ad7f49b6ece25cc19e5e7ec6e923d2f6c9a67c5d9c7063100a3df145b7da6a9"}
Oct 31 01:53:18.681813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.681789    5293 generic.go:356] "Generic (PLEG): container finished" podID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerID="8a77c40754a1f835eb94a54a4b38f7c7eb39d9bf4f087f3701c4e8c957c15138" exitCode=0
Oct 31 01:53:18.681928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.681831    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" event={"ID":"e4f874ec-7f56-4756-8959-fe8913ef679c","Type":"ContainerDied","Data":"8a77c40754a1f835eb94a54a4b38f7c7eb39d9bf4f087f3701c4e8c957c15138"}
Oct 31 01:53:18.681928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.681894    5293 scope.go:122] "RemoveContainer" containerID="52a3aff7a784858b06f9d82ee96df3fd7d8e1b97502fbb59b8a607e71c206266"
Oct 31 01:53:18.741134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:18.741062    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz" podStartSLOduration=2.741021913 podStartE2EDuration="2.741021913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:18.693177057 +0000 UTC m=+1956.641825450" watchObservedRunningTime="2026-10-31 01:53:18.741021913 +0000 UTC m=+1956.689670302"
Oct 31 01:53:19.142569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.142540    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.192494    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-certs\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.192564    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-podinfo\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.192652    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-socket\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.192750    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-envoy\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.192942    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-kube-api-access-t98gn\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-kube-api-access-t98gn\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.192972    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f874ec-7f56-4756-8959-fe8913ef679c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f874ec-7f56-4756-8959-fe8913ef679c-istiod-ca-cert\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.193093    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-data\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.193157    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-credential-socket\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.193193    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-token\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-token\") pod \"e4f874ec-7f56-4756-8959-fe8913ef679c\" (UID: \"e4f874ec-7f56-4756-8959-fe8913ef679c\") "
Oct 31 01:53:19.197641 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.197253    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-certs" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:19.250337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.200293    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-data" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:19.250337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.200619    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f874ec-7f56-4756-8959-fe8913ef679c-istiod-ca-cert" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:19.250337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.200860    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-credential-socket" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:19.250337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.236844    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-socket" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:19.321065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.307603    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f874ec-7f56-4756-8959-fe8913ef679c-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.321065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.307638    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.321065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.307651    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.321065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.307665    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.321065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.307677    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.357852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.357661    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-envoy" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:19.415237 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.409825    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.432280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.432233    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-kube-api-access-t98gn" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "kube-api-access-t98gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:19.460012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.457288    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-token" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:19.460012 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.459942    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-podinfo" pod "e4f874ec-7f56-4756-8959-fe8913ef679c" (UID: "e4f874ec-7f56-4756-8959-fe8913ef679c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:53:19.467536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.466724    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"]
Oct 31 01:53:19.467719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.467689    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy"
Oct 31 01:53:19.467719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.467708    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy"
Oct 31 01:53:19.467834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.467726    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy"
Oct 31 01:53:19.467834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.467734    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy"
Oct 31 01:53:19.467956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.467933    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy"
Oct 31 01:53:19.467956 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.467952    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" containerName="istio-proxy"
Oct 31 01:53:19.474088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.472213    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"]
Oct 31 01:53:19.474088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.472313    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.495026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.495004    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 01:53:19.495198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.495181    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 01:53:19.535199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.521097    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdgcg\" (UniqueName: \"kubernetes.io/projected/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-kube-api-access-jdgcg\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.535199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.521305    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.535199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.521336    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.535199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.521422    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98gn\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-kube-api-access-t98gn\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.535199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.521438    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.535199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.521452    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f874ec-7f56-4756-8959-fe8913ef679c-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:19.629112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.624068    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdgcg\" (UniqueName: \"kubernetes.io/projected/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-kube-api-access-jdgcg\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.629112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.624297    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.629112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.624328    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.629112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.625839    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.633346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.633316    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.639866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.639835    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdgcg\" (UniqueName: \"kubernetes.io/projected/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-kube-api-access-jdgcg\") pod \"maas-api-revoke-sub-2e73624a-k8jbw\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.692583 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.692553    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"
Oct 31 01:53:19.693007 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.692593    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4" event={"ID":"e4f874ec-7f56-4756-8959-fe8913ef679c","Type":"ContainerDied","Data":"0a8b99414d9f7542be251255307d86c09b853689bf7e78a2c628b81e44e2a22f"}
Oct 31 01:53:19.693007 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.692651    5293 scope.go:122] "RemoveContainer" containerID="8a77c40754a1f835eb94a54a4b38f7c7eb39d9bf4f087f3701c4e8c957c15138"
Oct 31 01:53:19.694719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.694696    5293 generic.go:356] "Generic (PLEG): container finished" podID="d06b6bc6-0460-4d28-85d5-eacbbe50a06c" containerID="feca8957f1d7d5e51f0d33a7b767326f96be8c7cefc761bbb140a688a1453801" exitCode=0
Oct 31 01:53:19.694830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.694781    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz" event={"ID":"d06b6bc6-0460-4d28-85d5-eacbbe50a06c","Type":"ContainerDied","Data":"feca8957f1d7d5e51f0d33a7b767326f96be8c7cefc761bbb140a688a1453801"}
Oct 31 01:53:19.825256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.799071    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:19.840065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.831939    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"]
Oct 31 01:53:19.840065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:19.838527    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-278a56-openshift-default-67447b54b5-74kf4"]
Oct 31 01:53:20.261118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.253340    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:20.285327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.285288    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f874ec-7f56-4756-8959-fe8913ef679c" path="/var/lib/kubelet/pods/e4f874ec-7f56-4756-8959-fe8913ef679c/volumes"
Oct 31 01:53:20.392728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.392658    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a4f410-7615-498e-bfc3-38297690ae6a-kube-api-access-jg5wl\" (UniqueName: \"kubernetes.io/projected/71a4f410-7615-498e-bfc3-38297690ae6a-kube-api-access-jg5wl\") pod \"71a4f410-7615-498e-bfc3-38297690ae6a\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") "
Oct 31 01:53:20.392895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.392732    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-openshift-ca\") pod \"71a4f410-7615-498e-bfc3-38297690ae6a\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") "
Oct 31 01:53:20.392895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.392819    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-xks-ca\") pod \"71a4f410-7615-498e-bfc3-38297690ae6a\" (UID: \"71a4f410-7615-498e-bfc3-38297690ae6a\") "
Oct 31 01:53:20.393451 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.393402    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-openshift-ca" pod "71a4f410-7615-498e-bfc3-38297690ae6a" (UID: "71a4f410-7615-498e-bfc3-38297690ae6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:20.393652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.393633    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:20.395771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.395743    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-xks-ca" pod "71a4f410-7615-498e-bfc3-38297690ae6a" (UID: "71a4f410-7615-498e-bfc3-38297690ae6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:20.395885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.395798    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a4f410-7615-498e-bfc3-38297690ae6a-kube-api-access-jg5wl" pod "71a4f410-7615-498e-bfc3-38297690ae6a" (UID: "71a4f410-7615-498e-bfc3-38297690ae6a"). InnerVolumeSpecName "kube-api-access-jg5wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:20.441733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.441706    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"]
Oct 31 01:53:20.444126 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:20.444095    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97dc99e_51dd_4ede_9b0d_9b7595b7d200.slice/crio-55d2b4d2b8fd1a8c137211c790ccd59cbe94316482af93a150f9ce75bb263422 WatchSource:0}: Error finding container 55d2b4d2b8fd1a8c137211c790ccd59cbe94316482af93a150f9ce75bb263422: Status 404 returned error can't find the container with id 55d2b4d2b8fd1a8c137211c790ccd59cbe94316482af93a150f9ce75bb263422
Oct 31 01:53:20.494690 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.494664    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg5wl\" (UniqueName: \"kubernetes.io/projected/71a4f410-7615-498e-bfc3-38297690ae6a-kube-api-access-jg5wl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:20.494792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.494690    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a4f410-7615-498e-bfc3-38297690ae6a-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:20.494952 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.494935    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 01:53:20.494998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.494975    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 01:53:20.725561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.725522    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw" event={"ID":"e97dc99e-51dd-4ede-9b0d-9b7595b7d200","Type":"ContainerStarted","Data":"55d2b4d2b8fd1a8c137211c790ccd59cbe94316482af93a150f9ce75bb263422"}
Oct 31 01:53:20.728703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.728616    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"
Oct 31 01:53:20.728878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.728620    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x" event={"ID":"71a4f410-7615-498e-bfc3-38297690ae6a","Type":"ContainerDied","Data":"b36062cf266446d5ac584f64e103e375c3dbbf55f36b941a0a1e58c14db3118f"}
Oct 31 01:53:20.729082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.729027    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b36062cf266446d5ac584f64e103e375c3dbbf55f36b941a0a1e58c14db3118f"
Oct 31 01:53:20.944188 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:20.944161    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:21.044116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.044007    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-xks-ca\") pod \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") "
Oct 31 01:53:21.044820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.044794    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-kube-api-access-wqgg5\" (UniqueName: \"kubernetes.io/projected/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-kube-api-access-wqgg5\") pod \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") "
Oct 31 01:53:21.045210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.045189    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-openshift-ca\") pod \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\" (UID: \"d06b6bc6-0460-4d28-85d5-eacbbe50a06c\") "
Oct 31 01:53:21.045561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.045535    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-openshift-ca" pod "d06b6bc6-0460-4d28-85d5-eacbbe50a06c" (UID: "d06b6bc6-0460-4d28-85d5-eacbbe50a06c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:21.046215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.046193    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:21.050241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.050203    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-kube-api-access-wqgg5" pod "d06b6bc6-0460-4d28-85d5-eacbbe50a06c" (UID: "d06b6bc6-0460-4d28-85d5-eacbbe50a06c"). InnerVolumeSpecName "kube-api-access-wqgg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:21.051648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.051619    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-xks-ca" pod "d06b6bc6-0460-4d28-85d5-eacbbe50a06c" (UID: "d06b6bc6-0460-4d28-85d5-eacbbe50a06c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:21.150626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.150591    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqgg5\" (UniqueName: \"kubernetes.io/projected/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-kube-api-access-wqgg5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:21.150626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.150627    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d06b6bc6-0460-4d28-85d5-eacbbe50a06c-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:21.522849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.522817    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:53:21.523378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.523347    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:53:21.524929 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.524889    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:53:21.741413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.741385    5293 generic.go:356] "Generic (PLEG): container finished" podID="e97dc99e-51dd-4ede-9b0d-9b7595b7d200" containerID="244079c16ba8ee4bf147692c4fb9b9782996d88209a5450b5bc59e045b5d28ef" exitCode=0
Oct 31 01:53:21.741773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.741468    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw" event={"ID":"e97dc99e-51dd-4ede-9b0d-9b7595b7d200","Type":"ContainerDied","Data":"244079c16ba8ee4bf147692c4fb9b9782996d88209a5450b5bc59e045b5d28ef"}
Oct 31 01:53:21.744088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.743355    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz" event={"ID":"d06b6bc6-0460-4d28-85d5-eacbbe50a06c","Type":"ContainerDied","Data":"0f42e87c9855b653152922dc1cd9186adb788756517db363e9efdf498de4f870"}
Oct 31 01:53:21.744088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.743377    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"
Oct 31 01:53:21.744088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:21.743414    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f42e87c9855b653152922dc1cd9186adb788756517db363e9efdf498de4f870"
Oct 31 01:53:23.005294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.005223    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:23.051116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.049943    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:53:23.051472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.051447    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:53:23.186129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.186094    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-xks-ca\") pod \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") "
Oct 31 01:53:23.186315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.186167    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-openshift-ca\") pod \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") "
Oct 31 01:53:23.186315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.186218    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-kube-api-access-jdgcg\" (UniqueName: \"kubernetes.io/projected/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-kube-api-access-jdgcg\") pod \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\" (UID: \"e97dc99e-51dd-4ede-9b0d-9b7595b7d200\") "
Oct 31 01:53:23.186531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.186507    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-openshift-ca" pod "e97dc99e-51dd-4ede-9b0d-9b7595b7d200" (UID: "e97dc99e-51dd-4ede-9b0d-9b7595b7d200"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:23.187391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.187367    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:23.188575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.188549    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-xks-ca" pod "e97dc99e-51dd-4ede-9b0d-9b7595b7d200" (UID: "e97dc99e-51dd-4ede-9b0d-9b7595b7d200"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:23.188672 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.188657    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-kube-api-access-jdgcg" pod "e97dc99e-51dd-4ede-9b0d-9b7595b7d200" (UID: "e97dc99e-51dd-4ede-9b0d-9b7595b7d200"). InnerVolumeSpecName "kube-api-access-jdgcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:23.288644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.288557    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:23.288644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.288590    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdgcg\" (UniqueName: \"kubernetes.io/projected/e97dc99e-51dd-4ede-9b0d-9b7595b7d200-kube-api-access-jdgcg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:23.760432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.760401    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"
Oct 31 01:53:23.760610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.760429    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw" event={"ID":"e97dc99e-51dd-4ede-9b0d-9b7595b7d200","Type":"ContainerDied","Data":"55d2b4d2b8fd1a8c137211c790ccd59cbe94316482af93a150f9ce75bb263422"}
Oct 31 01:53:23.760610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:23.760479    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d2b4d2b8fd1a8c137211c790ccd59cbe94316482af93a150f9ce75bb263422"
Oct 31 01:53:26.891704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:26.891650    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk" podUID="2f6490bd-3a92-467b-9c0e-346efcc86a5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:53:29.048129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.048093    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"]
Oct 31 01:53:29.049081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.048991    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs" podUID="8351577d-def3-46f5-9ccd-30772fa5a5c9" containerName="payload-processing" containerID="cri-o://adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36" gracePeriod=30
Oct 31 01:53:29.052984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.052958    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"]
Oct 31 01:53:29.053236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.053189    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr" podUID="2bab320c-4018-44c6-a7cf-06f76f5f315a" containerName="payload-pre-processing" containerID="cri-o://f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a" gracePeriod=30
Oct 31 01:53:29.069639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.069611    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"]
Oct 31 01:53:29.079278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.079247    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a779b3-hwqgc"]
Oct 31 01:53:29.397380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.397355    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:53:29.401603 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.401577    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:53:29.527065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.521325    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bab320c-4018-44c6-a7cf-06f76f5f315a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bab320c-4018-44c6-a7cf-06f76f5f315a-plugins-config-volume\") pod \"2bab320c-4018-44c6-a7cf-06f76f5f315a\" (UID: \"2bab320c-4018-44c6-a7cf-06f76f5f315a\") "
Oct 31 01:53:29.527065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.521389    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bab320c-4018-44c6-a7cf-06f76f5f315a-kube-api-access-zfxg9\" (UniqueName: \"kubernetes.io/projected/2bab320c-4018-44c6-a7cf-06f76f5f315a-kube-api-access-zfxg9\") pod \"2bab320c-4018-44c6-a7cf-06f76f5f315a\" (UID: \"2bab320c-4018-44c6-a7cf-06f76f5f315a\") "
Oct 31 01:53:29.527065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.521475    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8351577d-def3-46f5-9ccd-30772fa5a5c9-kube-api-access-dkfl8\" (UniqueName: \"kubernetes.io/projected/8351577d-def3-46f5-9ccd-30772fa5a5c9-kube-api-access-dkfl8\") pod \"8351577d-def3-46f5-9ccd-30772fa5a5c9\" (UID: \"8351577d-def3-46f5-9ccd-30772fa5a5c9\") "
Oct 31 01:53:29.527065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.521607    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8351577d-def3-46f5-9ccd-30772fa5a5c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8351577d-def3-46f5-9ccd-30772fa5a5c9-plugins-config-volume\") pod \"8351577d-def3-46f5-9ccd-30772fa5a5c9\" (UID: \"8351577d-def3-46f5-9ccd-30772fa5a5c9\") "
Oct 31 01:53:29.527065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.523901    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8351577d-def3-46f5-9ccd-30772fa5a5c9-plugins-config-volume" pod "8351577d-def3-46f5-9ccd-30772fa5a5c9" (UID: "8351577d-def3-46f5-9ccd-30772fa5a5c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:29.527065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.524393    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bab320c-4018-44c6-a7cf-06f76f5f315a-plugins-config-volume" pod "2bab320c-4018-44c6-a7cf-06f76f5f315a" (UID: "2bab320c-4018-44c6-a7cf-06f76f5f315a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:29.548066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.534492    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bab320c-4018-44c6-a7cf-06f76f5f315a-kube-api-access-zfxg9" pod "2bab320c-4018-44c6-a7cf-06f76f5f315a" (UID: "2bab320c-4018-44c6-a7cf-06f76f5f315a"). InnerVolumeSpecName "kube-api-access-zfxg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:29.584065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.583702    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8351577d-def3-46f5-9ccd-30772fa5a5c9-kube-api-access-dkfl8" pod "8351577d-def3-46f5-9ccd-30772fa5a5c9" (UID: "8351577d-def3-46f5-9ccd-30772fa5a5c9"). InnerVolumeSpecName "kube-api-access-dkfl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:29.624116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.623825    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bab320c-4018-44c6-a7cf-06f76f5f315a-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:29.624116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.623857    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfxg9\" (UniqueName: \"kubernetes.io/projected/2bab320c-4018-44c6-a7cf-06f76f5f315a-kube-api-access-zfxg9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:29.624116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.623867    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfl8\" (UniqueName: \"kubernetes.io/projected/8351577d-def3-46f5-9ccd-30772fa5a5c9-kube-api-access-dkfl8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:29.624116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.623876    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8351577d-def3-46f5-9ccd-30772fa5a5c9-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:29.746885 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.746848    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"]
Oct 31 01:53:29.750131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.750105    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3ff0c-5j74x"]
Oct 31 01:53:29.902664 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.901156    5293 generic.go:356] "Generic (PLEG): container finished" podID="2bab320c-4018-44c6-a7cf-06f76f5f315a" containerID="f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a" exitCode=0
Oct 31 01:53:29.902664 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.901565    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"
Oct 31 01:53:29.903224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.903196    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr" event={"ID":"2bab320c-4018-44c6-a7cf-06f76f5f315a","Type":"ContainerDied","Data":"f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a"}
Oct 31 01:53:29.903444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.903424    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr" event={"ID":"2bab320c-4018-44c6-a7cf-06f76f5f315a","Type":"ContainerDied","Data":"37f951397d999d36ef855a0b3dcc6a59e8d75e293f5553d4819a1ed8d2faf926"}
Oct 31 01:53:29.903577 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.903564    5293 scope.go:122] "RemoveContainer" containerID="f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a"
Oct 31 01:53:29.908705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.908679    5293 generic.go:356] "Generic (PLEG): container finished" podID="8351577d-def3-46f5-9ccd-30772fa5a5c9" containerID="adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36" exitCode=0
Oct 31 01:53:29.908966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.908947    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs" event={"ID":"8351577d-def3-46f5-9ccd-30772fa5a5c9","Type":"ContainerDied","Data":"adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36"}
Oct 31 01:53:29.909201 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.909178    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs" event={"ID":"8351577d-def3-46f5-9ccd-30772fa5a5c9","Type":"ContainerDied","Data":"7d69388aaa0e28c0984484243a42ccac0f266c3f05a67be5d378152fd7d36f54"}
Oct 31 01:53:29.909473 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.909454    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"
Oct 31 01:53:29.922742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.922724    5293 scope.go:122] "RemoveContainer" containerID="f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a"
Oct 31 01:53:29.923135 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:53:29.923112    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a\": container with ID starting with f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a not found: ID does not exist" containerID="f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a"
Oct 31 01:53:29.923294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.923261    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a"} err="failed to get container status \"f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a\": rpc error: code = NotFound desc = could not find container \"f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a\": container with ID starting with f5eb60b6872471c5a47cd6ca60a45df10b011c11375446615f2d32ba112e662a not found: ID does not exist"
Oct 31 01:53:29.923294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.923294    5293 scope.go:122] "RemoveContainer" containerID="adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36"
Oct 31 01:53:29.942083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.937221    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"]
Oct 31 01:53:29.945391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.945356    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a3ff0c-6f5899b7bc-db4vr"]
Oct 31 01:53:29.947470 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.947450    5293 scope.go:122] "RemoveContainer" containerID="adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36"
Oct 31 01:53:29.947854 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:53:29.947830    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36\": container with ID starting with adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36 not found: ID does not exist" containerID="adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36"
Oct 31 01:53:29.947940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.947870    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36"} err="failed to get container status \"adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36\": rpc error: code = NotFound desc = could not find container \"adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36\": container with ID starting with adbbe731c78c3d76b3ed97065b9f0ad8655422b18669130c2bb1d42a34e6bb36 not found: ID does not exist"
Oct 31 01:53:29.955366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.955345    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"]
Oct 31 01:53:29.961553 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:29.961531    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a3ff0c-5f649767d8-gq9xs"]
Oct 31 01:53:30.174369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.174298    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bab320c-4018-44c6-a7cf-06f76f5f315a" path="/var/lib/kubelet/pods/2bab320c-4018-44c6-a7cf-06f76f5f315a/volumes"
Oct 31 01:53:30.174775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.174760    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a4f410-7615-498e-bfc3-38297690ae6a" path="/var/lib/kubelet/pods/71a4f410-7615-498e-bfc3-38297690ae6a/volumes"
Oct 31 01:53:30.175186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.175171    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8351577d-def3-46f5-9ccd-30772fa5a5c9" path="/var/lib/kubelet/pods/8351577d-def3-46f5-9ccd-30772fa5a5c9/volumes"
Oct 31 01:53:30.175867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.175852    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6698e9-d63b-4aa4-9210-50d9560fa3c6" path="/var/lib/kubelet/pods/be6698e9-d63b-4aa4-9210-50d9560fa3c6/volumes"
Oct 31 01:53:30.347590 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.347561    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:53:30.849822 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.849789    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"]
Oct 31 01:53:30.850695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850671    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e97dc99e-51dd-4ede-9b0d-9b7595b7d200" containerName="revoke-keys"
Oct 31 01:53:30.850695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850694    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97dc99e-51dd-4ede-9b0d-9b7595b7d200" containerName="revoke-keys"
Oct 31 01:53:30.850868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850715    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8351577d-def3-46f5-9ccd-30772fa5a5c9" containerName="payload-processing"
Oct 31 01:53:30.850868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850720    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8351577d-def3-46f5-9ccd-30772fa5a5c9" containerName="payload-processing"
Oct 31 01:53:30.850868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850745    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2bab320c-4018-44c6-a7cf-06f76f5f315a" containerName="payload-pre-processing"
Oct 31 01:53:30.850868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850754    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bab320c-4018-44c6-a7cf-06f76f5f315a" containerName="payload-pre-processing"
Oct 31 01:53:30.850868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850772    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d06b6bc6-0460-4d28-85d5-eacbbe50a06c" containerName="revoke-keys"
Oct 31 01:53:30.850868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850777    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b6bc6-0460-4d28-85d5-eacbbe50a06c" containerName="revoke-keys"
Oct 31 01:53:30.850868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850789    5293 cpu_manager.go:414] "containerMap: removing container" podUID="71a4f410-7615-498e-bfc3-38297690ae6a" containerName="revoke-keys"
Oct 31 01:53:30.850868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850794    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a4f410-7615-498e-bfc3-38297690ae6a" containerName="revoke-keys"
Oct 31 01:53:30.851229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850918    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bab320c-4018-44c6-a7cf-06f76f5f315a" containerName="payload-pre-processing"
Oct 31 01:53:30.851229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850932    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b6bc6-0460-4d28-85d5-eacbbe50a06c" containerName="revoke-keys"
Oct 31 01:53:30.851229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850939    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a4f410-7615-498e-bfc3-38297690ae6a" containerName="revoke-keys"
Oct 31 01:53:30.851229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850947    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97dc99e-51dd-4ede-9b0d-9b7595b7d200" containerName="revoke-keys"
Oct 31 01:53:30.851229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.850954    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8351577d-def3-46f5-9ccd-30772fa5a5c9" containerName="payload-processing"
Oct 31 01:53:30.856579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.856559    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.862205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.862186    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-77e2b5-openshift-default-dockercfg-9scxf\""
Oct 31 01:53:30.890447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.890417    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"]
Oct 31 01:53:30.974970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.974934    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-envoy\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.975196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.975072    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-socket\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.975196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.975105    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493b0f59-1286-4e4c-a47e-871418369fd9-istiod-ca-cert\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.975305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.975191    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-credential-socket\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.975305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.975260    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-certs\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.975398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.975360    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsnj\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-kube-api-access-zhsnj\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.975450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.975432    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-data\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.975534 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.975522    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-istio-token\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:30.975567 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:30.975542    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493b0f59-1286-4e4c-a47e-871418369fd9-istio-podinfo\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079214    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-credential-socket\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-certs\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079476    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsnj\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-kube-api-access-zhsnj\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079529    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-data\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079671    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-istio-token\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079702    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493b0f59-1286-4e4c-a47e-871418369fd9-istio-podinfo\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079805    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-envoy\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079858    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-socket\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.079885    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493b0f59-1286-4e4c-a47e-871418369fd9-istiod-ca-cert\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.080725    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493b0f59-1286-4e4c-a47e-871418369fd9-istiod-ca-cert\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.080982    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-credential-socket\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.081222    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-certs\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.084021 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.081801    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-data\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.100127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.097291    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-socket\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.126674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.126638    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493b0f59-1286-4e4c-a47e-871418369fd9-istio-podinfo\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.148079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.137459    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsnj\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-kube-api-access-zhsnj\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.355111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.354427    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-envoy\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.363792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.363719    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-istio-token\") pod \"e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") " pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.480305 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.480271    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"]
Oct 31 01:53:31.494066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.492361    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:31.519335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.519273    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"]
Oct 31 01:53:31.519500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.519407    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.523338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.523315    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:31.523580 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.523566    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:53:31.523813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.523798    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:31.523999 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.523986    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:31.651360 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.651279    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1948dd27-a949-499c-a849-1cee19c64633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.662224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.662120    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4lfm\" (UniqueName: \"kubernetes.io/projected/1948dd27-a949-499c-a849-1cee19c64633-kube-api-access-d4lfm\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.662474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.662455    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1948dd27-a949-499c-a849-1cee19c64633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.765917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.765007    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1948dd27-a949-499c-a849-1cee19c64633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.765917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.765121    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4lfm\" (UniqueName: \"kubernetes.io/projected/1948dd27-a949-499c-a849-1cee19c64633-kube-api-access-d4lfm\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.765917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.765151    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1948dd27-a949-499c-a849-1cee19c64633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.767099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.767070    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1948dd27-a949-499c-a849-1cee19c64633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.845959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.832301    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1948dd27-a949-499c-a849-1cee19c64633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.845959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.842858    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4lfm\" (UniqueName: \"kubernetes.io/projected/1948dd27-a949-499c-a849-1cee19c64633-kube-api-access-d4lfm\") pod \"maas-api-revoke-sub-3a3bf7b9-m92h8\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:31.960124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:31.925596    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:32.495504 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:32.495469    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"]
Oct 31 01:53:32.597546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:32.597498    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:32.597783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:32.597757    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:32.597968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:32.597946    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:33.511068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:33.510315    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"]
Oct 31 01:53:33.558963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:33.558896    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" event={"ID":"493b0f59-1286-4e4c-a47e-871418369fd9","Type":"ContainerStarted","Data":"9164740a609817d6716a6b7ffc1b7995a26d88b75a169ce4ab0ecaf1ac73523a"}
Oct 31 01:53:33.787230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:33.787125    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:33.787862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:33.787484    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 31 01:53:33.808157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:33.808118    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:33.855179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:33.855131    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:33.950197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:33.950149    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.030095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.029859    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.108071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.107955    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.152228 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.152175    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.281258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.281210    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.290166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.290121    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.814064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.813144    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.890315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.878389    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:53:34.891366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.891315    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:53:34.984473 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.984329    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded"
Oct 31 01:53:34.985360 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.984844    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:53:34.990651 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:34.990614    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:53:35.072282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:35.032234    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:53:35.081901 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:35.074857    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" event={"ID":"493b0f59-1286-4e4c-a47e-871418369fd9","Type":"ContainerStarted","Data":"93c574bba911cde51456888ecfd597691ad14e0cb5e178fd68713ae896e86112"}
Oct 31 01:53:35.081901 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:35.081662    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:53:35.087622 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:35.087578    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8" event={"ID":"1948dd27-a949-499c-a849-1cee19c64633","Type":"ContainerStarted","Data":"1195d1dc5173a62c8a8ae615ffa962b40417c3184a7c3a7a35ca418300fee096"}
Oct 31 01:53:35.087769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:35.087645    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8" event={"ID":"1948dd27-a949-499c-a849-1cee19c64633","Type":"ContainerStarted","Data":"5392f86c10e9edb6a61cd493abb648229d5a41211919864403014bcbc4da2b83"}
Oct 31 01:53:35.140605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:35.140556    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podStartSLOduration=5.140542062 podStartE2EDuration="5.140542062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:35.138442469 +0000 UTC m=+1973.087090861" watchObservedRunningTime="2026-10-31 01:53:35.140542062 +0000 UTC m=+1973.089190486"
Oct 31 01:53:35.492502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:35.492463    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:35.498275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:35.498249    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:36.103071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.100211    5293 generic.go:356] "Generic (PLEG): container finished" podID="1948dd27-a949-499c-a849-1cee19c64633" containerID="1195d1dc5173a62c8a8ae615ffa962b40417c3184a7c3a7a35ca418300fee096" exitCode=0
Oct 31 01:53:36.103071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.100256    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8" event={"ID":"1948dd27-a949-499c-a849-1cee19c64633","Type":"ContainerDied","Data":"1195d1dc5173a62c8a8ae615ffa962b40417c3184a7c3a7a35ca418300fee096"}
Oct 31 01:53:36.103071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.101293    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:36.103071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.102942    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:53:36.439410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.439382    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:36.603405 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.603375    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:53:36.631741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.631709    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1948dd27-a949-499c-a849-1cee19c64633-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1948dd27-a949-499c-a849-1cee19c64633-maas-api-xks-ca\") pod \"1948dd27-a949-499c-a849-1cee19c64633\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") "
Oct 31 01:53:36.631922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.631848    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1948dd27-a949-499c-a849-1cee19c64633-kube-api-access-d4lfm\" (UniqueName: \"kubernetes.io/projected/1948dd27-a949-499c-a849-1cee19c64633-kube-api-access-d4lfm\") pod \"1948dd27-a949-499c-a849-1cee19c64633\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") "
Oct 31 01:53:36.631922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.631901    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1948dd27-a949-499c-a849-1cee19c64633-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1948dd27-a949-499c-a849-1cee19c64633-maas-api-openshift-ca\") pod \"1948dd27-a949-499c-a849-1cee19c64633\" (UID: \"1948dd27-a949-499c-a849-1cee19c64633\") "
Oct 31 01:53:36.632562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.632529    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1948dd27-a949-499c-a849-1cee19c64633-maas-api-openshift-ca" pod "1948dd27-a949-499c-a849-1cee19c64633" (UID: "1948dd27-a949-499c-a849-1cee19c64633"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:36.634607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.634580    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1948dd27-a949-499c-a849-1cee19c64633-maas-api-xks-ca" pod "1948dd27-a949-499c-a849-1cee19c64633" (UID: "1948dd27-a949-499c-a849-1cee19c64633"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:36.634892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.634864    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1948dd27-a949-499c-a849-1cee19c64633-kube-api-access-d4lfm" pod "1948dd27-a949-499c-a849-1cee19c64633" (UID: "1948dd27-a949-499c-a849-1cee19c64633"). InnerVolumeSpecName "kube-api-access-d4lfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:36.733061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.733018    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6490bd-3a92-467b-9c0e-346efcc86a5a-kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/2f6490bd-3a92-467b-9c0e-346efcc86a5a-kube-api-access-nv84g\") pod \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\" (UID: \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\") "
Oct 31 01:53:36.733212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.733121    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6490bd-3a92-467b-9c0e-346efcc86a5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6490bd-3a92-467b-9c0e-346efcc86a5a-plugins-config-volume\") pod \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\" (UID: \"2f6490bd-3a92-467b-9c0e-346efcc86a5a\") "
Oct 31 01:53:36.733538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.733516    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6490bd-3a92-467b-9c0e-346efcc86a5a-plugins-config-volume" pod "2f6490bd-3a92-467b-9c0e-346efcc86a5a" (UID: "2f6490bd-3a92-467b-9c0e-346efcc86a5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:36.733689 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.733674    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6490bd-3a92-467b-9c0e-346efcc86a5a-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:36.733951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.733692    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1948dd27-a949-499c-a849-1cee19c64633-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:36.733951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.733702    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4lfm\" (UniqueName: \"kubernetes.io/projected/1948dd27-a949-499c-a849-1cee19c64633-kube-api-access-d4lfm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:36.733951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.733711    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1948dd27-a949-499c-a849-1cee19c64633-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:36.735585 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.735556    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6490bd-3a92-467b-9c0e-346efcc86a5a-kube-api-access-nv84g" pod "2f6490bd-3a92-467b-9c0e-346efcc86a5a" (UID: "2f6490bd-3a92-467b-9c0e-346efcc86a5a"). InnerVolumeSpecName "kube-api-access-nv84g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:36.834998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:36.834960    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/2f6490bd-3a92-467b-9c0e-346efcc86a5a-kube-api-access-nv84g\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:37.199101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.198222    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8" event={"ID":"1948dd27-a949-499c-a849-1cee19c64633","Type":"ContainerDied","Data":"5392f86c10e9edb6a61cd493abb648229d5a41211919864403014bcbc4da2b83"}
Oct 31 01:53:37.199101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.198317    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5392f86c10e9edb6a61cd493abb648229d5a41211919864403014bcbc4da2b83"
Oct 31 01:53:37.199101 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.198406    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"
Oct 31 01:53:37.248991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.240147    5293 generic.go:356] "Generic (PLEG): container finished" podID="2f6490bd-3a92-467b-9c0e-346efcc86a5a" containerID="8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883" exitCode=0
Oct 31 01:53:37.248991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.240565    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"
Oct 31 01:53:37.248991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.240615    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk" event={"ID":"2f6490bd-3a92-467b-9c0e-346efcc86a5a","Type":"ContainerDied","Data":"8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883"}
Oct 31 01:53:37.248991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.240665    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk" event={"ID":"2f6490bd-3a92-467b-9c0e-346efcc86a5a","Type":"ContainerDied","Data":"2e1a97455ffae01aa6dc5176a138d2b1d654e59b9242673d75bebf70d57fbb66"}
Oct 31 01:53:37.248991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.240700    5293 scope.go:122] "RemoveContainer" containerID="8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883"
Oct 31 01:53:37.282408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.282386    5293 scope.go:122] "RemoveContainer" containerID="8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883"
Oct 31 01:53:37.283204 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:53:37.283173    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883\": container with ID starting with 8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883 not found: ID does not exist" containerID="8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883"
Oct 31 01:53:37.283311 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.283215    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883"} err="failed to get container status \"8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883\": rpc error: code = NotFound desc = could not find container \"8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883\": container with ID starting with 8d505d082e545d60e26ba0ec7dc511ae78e39238c7254ab20931c79f47991883 not found: ID does not exist"
Oct 31 01:53:37.297294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.297261    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"]
Oct 31 01:53:37.301460 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:37.301435    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a779b3-55f94f74f7-lckxk"]
Oct 31 01:53:38.252145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.251896    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6490bd-3a92-467b-9c0e-346efcc86a5a" path="/var/lib/kubelet/pods/2f6490bd-3a92-467b-9c0e-346efcc86a5a/volumes"
Oct 31 01:53:38.880443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.880409    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"]
Oct 31 01:53:38.881072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.881034    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2f6490bd-3a92-467b-9c0e-346efcc86a5a" containerName="payload-processing"
Oct 31 01:53:38.881072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.881065    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6490bd-3a92-467b-9c0e-346efcc86a5a" containerName="payload-processing"
Oct 31 01:53:38.881175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.881074    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1948dd27-a949-499c-a849-1cee19c64633" containerName="revoke-keys"
Oct 31 01:53:38.881175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.881080    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1948dd27-a949-499c-a849-1cee19c64633" containerName="revoke-keys"
Oct 31 01:53:38.881235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.881174    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6490bd-3a92-467b-9c0e-346efcc86a5a" containerName="payload-processing"
Oct 31 01:53:38.881235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.881184    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1948dd27-a949-499c-a849-1cee19c64633" containerName="revoke-keys"
Oct 31 01:53:38.885984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.885965    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:38.890269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.890017    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-77e2b5\""
Oct 31 01:53:38.890269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.890188    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-77e2b5-dockercfg-wx72g\""
Oct 31 01:53:38.895484 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.895464    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"]
Oct 31 01:53:38.930023 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.929973    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cba5706-9923-4e46-a3bf-60b97650625c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw\" (UID: \"8cba5706-9923-4e46-a3bf-60b97650625c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:38.930310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.930151    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thkn6\" (UniqueName: \"kubernetes.io/projected/8cba5706-9923-4e46-a3bf-60b97650625c-kube-api-access-thkn6\") pod \"payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw\" (UID: \"8cba5706-9923-4e46-a3bf-60b97650625c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:38.947111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.947075    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"]
Oct 31 01:53:38.953540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.953518    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:38.960348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:38.960324    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"]
Oct 31 01:53:39.055064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.036643    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thkn6\" (UniqueName: \"kubernetes.io/projected/8cba5706-9923-4e46-a3bf-60b97650625c-kube-api-access-thkn6\") pod \"payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw\" (UID: \"8cba5706-9923-4e46-a3bf-60b97650625c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:39.055064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.036906    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cba5706-9923-4e46-a3bf-60b97650625c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw\" (UID: \"8cba5706-9923-4e46-a3bf-60b97650625c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:39.055064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.037899    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cba5706-9923-4e46-a3bf-60b97650625c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw\" (UID: \"8cba5706-9923-4e46-a3bf-60b97650625c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:39.182066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.174016    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thkn6\" (UniqueName: \"kubernetes.io/projected/8cba5706-9923-4e46-a3bf-60b97650625c-kube-api-access-thkn6\") pod \"payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw\" (UID: \"8cba5706-9923-4e46-a3bf-60b97650625c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:39.209070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.205336    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:39.209070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.168736    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5817f1-a01d-4951-a3c3-f0e1717a738f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz\" (UID: \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:39.209070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.206735    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8j4\" (UniqueName: \"kubernetes.io/projected/8d5817f1-a01d-4951-a3c3-f0e1717a738f-kube-api-access-4b8j4\") pod \"payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz\" (UID: \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:39.246104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.230325    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"]
Oct 31 01:53:39.283778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.270158    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"]
Oct 31 01:53:39.283778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.270288    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.283778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.276806    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:53:39.283778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.280099    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:39.283778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.280158    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:39.283778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.280237    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:39.345726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.345306    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7gs2\" (UniqueName: \"kubernetes.io/projected/cf27db1d-91ff-4e40-9115-25b718b0a88f-kube-api-access-m7gs2\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.345726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.345462    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5817f1-a01d-4951-a3c3-f0e1717a738f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz\" (UID: \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:39.345726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.345663    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8j4\" (UniqueName: \"kubernetes.io/projected/8d5817f1-a01d-4951-a3c3-f0e1717a738f-kube-api-access-4b8j4\") pod \"payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz\" (UID: \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:39.346025 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.345835    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.346025 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.345991    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.354029 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.352931    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5817f1-a01d-4951-a3c3-f0e1717a738f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz\" (UID: \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:39.361646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.360432    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8j4\" (UniqueName: \"kubernetes.io/projected/8d5817f1-a01d-4951-a3c3-f0e1717a738f-kube-api-access-4b8j4\") pod \"payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz\" (UID: \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:39.448200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.448106    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7gs2\" (UniqueName: \"kubernetes.io/projected/cf27db1d-91ff-4e40-9115-25b718b0a88f-kube-api-access-m7gs2\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.448363 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.448343    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.448417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.448378    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.449189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.449165    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.451002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.450982    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.460432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.460408    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7gs2\" (UniqueName: \"kubernetes.io/projected/cf27db1d-91ff-4e40-9115-25b718b0a88f-kube-api-access-m7gs2\") pod \"maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.571128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.571089    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:39.607653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.607610    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"]
Oct 31 01:53:39.612616 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:39.612588    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cba5706_9923_4e46_a3bf_60b97650625c.slice/crio-c39ae8464b76b065d150c41832763db078d3d06b9c7e32122a34ef4d817b45ed WatchSource:0}: Error finding container c39ae8464b76b065d150c41832763db078d3d06b9c7e32122a34ef4d817b45ed: Status 404 returned error can't find the container with id c39ae8464b76b065d150c41832763db078d3d06b9c7e32122a34ef4d817b45ed
Oct 31 01:53:39.666062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.666008    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:39.925710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.925672    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"]
Oct 31 01:53:39.932498 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:39.932458    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5817f1_a01d_4951_a3c3_f0e1717a738f.slice/crio-10392b5c635f62ba34f34f3927c6d49d9fbb1b5fcf2eb0ec7602b8118533cded WatchSource:0}: Error finding container 10392b5c635f62ba34f34f3927c6d49d9fbb1b5fcf2eb0ec7602b8118533cded: Status 404 returned error can't find the container with id 10392b5c635f62ba34f34f3927c6d49d9fbb1b5fcf2eb0ec7602b8118533cded
Oct 31 01:53:39.972104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:39.972079    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"]
Oct 31 01:53:39.975180 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:39.975154    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf27db1d_91ff_4e40_9115_25b718b0a88f.slice/crio-65c9491d1740b0a364fc20999fd2d1781f74cba6b15c4ec1f92c05ea1817d84a WatchSource:0}: Error finding container 65c9491d1740b0a364fc20999fd2d1781f74cba6b15c4ec1f92c05ea1817d84a: Status 404 returned error can't find the container with id 65c9491d1740b0a364fc20999fd2d1781f74cba6b15c4ec1f92c05ea1817d84a
Oct 31 01:53:40.343291 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.343258    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz" event={"ID":"8d5817f1-a01d-4951-a3c3-f0e1717a738f","Type":"ContainerStarted","Data":"d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489"}
Oct 31 01:53:40.343759 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.343320    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz" event={"ID":"8d5817f1-a01d-4951-a3c3-f0e1717a738f","Type":"ContainerStarted","Data":"10392b5c635f62ba34f34f3927c6d49d9fbb1b5fcf2eb0ec7602b8118533cded"}
Oct 31 01:53:40.343759 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.343558    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:40.345278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.345247    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw" event={"ID":"8cba5706-9923-4e46-a3bf-60b97650625c","Type":"ContainerStarted","Data":"c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf"}
Oct 31 01:53:40.345436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.345302    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw" event={"ID":"8cba5706-9923-4e46-a3bf-60b97650625c","Type":"ContainerStarted","Data":"c39ae8464b76b065d150c41832763db078d3d06b9c7e32122a34ef4d817b45ed"}
Oct 31 01:53:40.345436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.345346    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:40.347133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.347107    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg" event={"ID":"cf27db1d-91ff-4e40-9115-25b718b0a88f","Type":"ContainerStarted","Data":"9adf1cb7a4d1bb37bfee81c800712f9dd4b817a89494f28195151f17142e381c"}
Oct 31 01:53:40.347235 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.347149    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg" event={"ID":"cf27db1d-91ff-4e40-9115-25b718b0a88f","Type":"ContainerStarted","Data":"65c9491d1740b0a364fc20999fd2d1781f74cba6b15c4ec1f92c05ea1817d84a"}
Oct 31 01:53:40.364548 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.364506    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz" podStartSLOduration=2.36449218 podStartE2EDuration="2.36449218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:40.360309779 +0000 UTC m=+1978.308958173" watchObservedRunningTime="2026-10-31 01:53:40.36449218 +0000 UTC m=+1978.313140568"
Oct 31 01:53:40.379445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.379380    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw" podStartSLOduration=2.379366444 podStartE2EDuration="2.379366444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:40.377795227 +0000 UTC m=+1978.326443619" watchObservedRunningTime="2026-10-31 01:53:40.379366444 +0000 UTC m=+1978.328014835"
Oct 31 01:53:40.394752 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.394713    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg" podStartSLOduration=1.3947009750000001 podStartE2EDuration="1.394700975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:40.394492285 +0000 UTC m=+1978.343140678" watchObservedRunningTime="2026-10-31 01:53:40.394700975 +0000 UTC m=+1978.343349366"
Oct 31 01:53:40.746334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.746301    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"]
Oct 31 01:53:40.750744 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.750726    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.761495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.761469    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"]
Oct 31 01:53:40.874481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.874445    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.874481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.874483    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glt52\" (UniqueName: \"kubernetes.io/projected/94f98a85-24fd-481a-b607-df3260ae9db8-kube-api-access-glt52\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.874684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.874646    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.975499 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.975470    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.975698 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.975511    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glt52\" (UniqueName: \"kubernetes.io/projected/94f98a85-24fd-481a-b607-df3260ae9db8-kube-api-access-glt52\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.975698 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.975679    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.976196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.976167    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.978329 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.978307    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:40.986921 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:40.986900    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glt52\" (UniqueName: \"kubernetes.io/projected/94f98a85-24fd-481a-b607-df3260ae9db8-kube-api-access-glt52\") pod \"maas-api-revoke-sub-65f40888-8mqzq\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:41.066391 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:41.066359    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:41.378113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:41.378009    5293 generic.go:356] "Generic (PLEG): container finished" podID="cf27db1d-91ff-4e40-9115-25b718b0a88f" containerID="9adf1cb7a4d1bb37bfee81c800712f9dd4b817a89494f28195151f17142e381c" exitCode=0
Oct 31 01:53:41.379513 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:41.378611    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg" event={"ID":"cf27db1d-91ff-4e40-9115-25b718b0a88f","Type":"ContainerDied","Data":"9adf1cb7a4d1bb37bfee81c800712f9dd4b817a89494f28195151f17142e381c"}
Oct 31 01:53:41.445375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:41.445345    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"]
Oct 31 01:53:41.779262 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:53:41.779215    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:53:42.389849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.389816    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq" event={"ID":"94f98a85-24fd-481a-b607-df3260ae9db8","Type":"ContainerStarted","Data":"906a26611aef4acf3d366fc1049e8e1ff88a1d0143e9bd1a94075f3edff8fdd3"}
Oct 31 01:53:42.390304 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.389873    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq" event={"ID":"94f98a85-24fd-481a-b607-df3260ae9db8","Type":"ContainerStarted","Data":"0bd2fa82aca7992616ce8b48a116b5532f90bf3a0506d45ccdaf7ada735c9b96"}
Oct 31 01:53:42.411368 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.411297    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq" podStartSLOduration=2.411277775 podStartE2EDuration="2.411277775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:42.406437486 +0000 UTC m=+1980.355085872" watchObservedRunningTime="2026-10-31 01:53:42.411277775 +0000 UTC m=+1980.359926166"
Oct 31 01:53:42.590989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.590963    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:42.623501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.623467    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-openshift-ca\") pod \"cf27db1d-91ff-4e40-9115-25b718b0a88f\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") "
Oct 31 01:53:42.623652 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.623598    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-xks-ca\") pod \"cf27db1d-91ff-4e40-9115-25b718b0a88f\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") "
Oct 31 01:53:42.623744 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.623728    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf27db1d-91ff-4e40-9115-25b718b0a88f-kube-api-access-m7gs2\" (UniqueName: \"kubernetes.io/projected/cf27db1d-91ff-4e40-9115-25b718b0a88f-kube-api-access-m7gs2\") pod \"cf27db1d-91ff-4e40-9115-25b718b0a88f\" (UID: \"cf27db1d-91ff-4e40-9115-25b718b0a88f\") "
Oct 31 01:53:42.623914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.623880    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-openshift-ca" pod "cf27db1d-91ff-4e40-9115-25b718b0a88f" (UID: "cf27db1d-91ff-4e40-9115-25b718b0a88f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:42.624464 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.624352    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:42.626020 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.625999    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-xks-ca" pod "cf27db1d-91ff-4e40-9115-25b718b0a88f" (UID: "cf27db1d-91ff-4e40-9115-25b718b0a88f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:42.626586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.626558    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf27db1d-91ff-4e40-9115-25b718b0a88f-kube-api-access-m7gs2" pod "cf27db1d-91ff-4e40-9115-25b718b0a88f" (UID: "cf27db1d-91ff-4e40-9115-25b718b0a88f"). InnerVolumeSpecName "kube-api-access-m7gs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:42.726183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.726147    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7gs2\" (UniqueName: \"kubernetes.io/projected/cf27db1d-91ff-4e40-9115-25b718b0a88f-kube-api-access-m7gs2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:42.726183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:42.726177    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf27db1d-91ff-4e40-9115-25b718b0a88f-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.400910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:43.400878    5293 generic.go:356] "Generic (PLEG): container finished" podID="94f98a85-24fd-481a-b607-df3260ae9db8" containerID="906a26611aef4acf3d366fc1049e8e1ff88a1d0143e9bd1a94075f3edff8fdd3" exitCode=0
Oct 31 01:53:43.401406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:43.400975    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq" event={"ID":"94f98a85-24fd-481a-b607-df3260ae9db8","Type":"ContainerDied","Data":"906a26611aef4acf3d366fc1049e8e1ff88a1d0143e9bd1a94075f3edff8fdd3"}
Oct 31 01:53:43.402897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:43.402860    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg" event={"ID":"cf27db1d-91ff-4e40-9115-25b718b0a88f","Type":"ContainerDied","Data":"65c9491d1740b0a364fc20999fd2d1781f74cba6b15c4ec1f92c05ea1817d84a"}
Oct 31 01:53:43.402897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:43.402893    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"
Oct 31 01:53:43.403107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:43.402919    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65c9491d1740b0a364fc20999fd2d1781f74cba6b15c4ec1f92c05ea1817d84a"
Oct 31 01:53:44.718804 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.718773    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:44.779015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.778983    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f98a85-24fd-481a-b607-df3260ae9db8-kube-api-access-glt52\" (UniqueName: \"kubernetes.io/projected/94f98a85-24fd-481a-b607-df3260ae9db8-kube-api-access-glt52\") pod \"94f98a85-24fd-481a-b607-df3260ae9db8\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") "
Oct 31 01:53:44.779256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.779101    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-openshift-ca\") pod \"94f98a85-24fd-481a-b607-df3260ae9db8\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") "
Oct 31 01:53:44.779256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.779136    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-xks-ca\") pod \"94f98a85-24fd-481a-b607-df3260ae9db8\" (UID: \"94f98a85-24fd-481a-b607-df3260ae9db8\") "
Oct 31 01:53:44.779500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.779474    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-openshift-ca" pod "94f98a85-24fd-481a-b607-df3260ae9db8" (UID: "94f98a85-24fd-481a-b607-df3260ae9db8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:44.779911 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.779883    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:44.783525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.783492    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-xks-ca" pod "94f98a85-24fd-481a-b607-df3260ae9db8" (UID: "94f98a85-24fd-481a-b607-df3260ae9db8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:44.783654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.783631    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f98a85-24fd-481a-b607-df3260ae9db8-kube-api-access-glt52" pod "94f98a85-24fd-481a-b607-df3260ae9db8" (UID: "94f98a85-24fd-481a-b607-df3260ae9db8"). InnerVolumeSpecName "kube-api-access-glt52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:44.885070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.882191    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glt52\" (UniqueName: \"kubernetes.io/projected/94f98a85-24fd-481a-b607-df3260ae9db8-kube-api-access-glt52\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:44.885070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:44.882233    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f98a85-24fd-481a-b607-df3260ae9db8-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:45.108667 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:45.108591    5293 scope.go:122] "RemoveContainer" containerID="f7bc424a0e4eb400dbfa1e09f6022ae05405c0b602ed8125520377ea0d73cbdd"
Oct 31 01:53:45.119942 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:45.119919    5293 scope.go:122] "RemoveContainer" containerID="8428f52e3630cb86be2a7f4fa0571625c1659c8896716b930a406ca0c6964888"
Oct 31 01:53:45.130438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:45.130412    5293 scope.go:122] "RemoveContainer" containerID="d76cbef5f4b865949226bc6ab2f70b1d48fb4bf30d9a91f4d80e9a2f32d168f7"
Oct 31 01:53:45.142689 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:45.142665    5293 scope.go:122] "RemoveContainer" containerID="1f9bddbb6746b92cb660aa2cff58923564a11f53bf83abe3567e67b4cb02578e"
Oct 31 01:53:45.153093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:45.153074    5293 scope.go:122] "RemoveContainer" containerID="4bac1898275522dfac13017733a492875827dfc006cdc16ca584ee64fabb69af"
Oct 31 01:53:45.436195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:45.436153    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq" event={"ID":"94f98a85-24fd-481a-b607-df3260ae9db8","Type":"ContainerDied","Data":"0bd2fa82aca7992616ce8b48a116b5532f90bf3a0506d45ccdaf7ada735c9b96"}
Oct 31 01:53:45.436402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:45.436197    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"
Oct 31 01:53:45.436402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:45.436204    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd2fa82aca7992616ce8b48a116b5532f90bf3a0506d45ccdaf7ada735c9b96"
Oct 31 01:53:51.383234 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:51.383196    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:53:51.383644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:51.383451    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:53:54.620147 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.620060    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"]
Oct 31 01:53:54.620929 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.620911    5293 cpu_manager.go:414] "containerMap: removing container" podUID="cf27db1d-91ff-4e40-9115-25b718b0a88f" containerName="revoke-keys"
Oct 31 01:53:54.620929 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.620931    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf27db1d-91ff-4e40-9115-25b718b0a88f" containerName="revoke-keys"
Oct 31 01:53:54.621106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.620943    5293 cpu_manager.go:414] "containerMap: removing container" podUID="94f98a85-24fd-481a-b607-df3260ae9db8" containerName="revoke-keys"
Oct 31 01:53:54.621106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.620948    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f98a85-24fd-481a-b607-df3260ae9db8" containerName="revoke-keys"
Oct 31 01:53:54.621215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.621139    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf27db1d-91ff-4e40-9115-25b718b0a88f" containerName="revoke-keys"
Oct 31 01:53:54.621215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.621160    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f98a85-24fd-481a-b607-df3260ae9db8" containerName="revoke-keys"
Oct 31 01:53:54.627425 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.627402    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.630757 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.628025    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"]
Oct 31 01:53:54.633351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.633329    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.652813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.644822    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"]
Oct 31 01:53:54.654913 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.654887    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"]
Oct 31 01:53:54.750017 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.749980    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7qc\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-kube-api-access-th7qc\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750017 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750022    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.750266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750174    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.750266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750208    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750288    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefe17a5-e46f-424f-82bf-b030fa88e212-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.750345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750317    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750381    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750411    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpj9\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-kube-api-access-wmpj9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.750491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750437    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.750491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750452    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750530    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750557    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750582    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.750626 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750618    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.750749 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750721    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.750784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750753    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b09ad6fd-125c-4f54-81c5-5862579af346-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750775    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b09ad6fd-125c-4f54-81c5-5862579af346-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.750871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.750858    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.835342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.835306    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:53:54.851794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.851762    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.851966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.851828    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7qc\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-kube-api-access-th7qc\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.851966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.851858    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.851966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.851920    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.851966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.851948    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.851995    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefe17a5-e46f-424f-82bf-b030fa88e212-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.852231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852022    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852113    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852163    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpj9\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-kube-api-access-wmpj9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.852231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852222    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852252    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852298    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852340    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852360    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852383    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852460    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852589    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852601    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852637    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b09ad6fd-125c-4f54-81c5-5862579af346-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852676    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b09ad6fd-125c-4f54-81c5-5862579af346-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.852771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852760    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.853982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852835    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefe17a5-e46f-424f-82bf-b030fa88e212-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.854131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.852962    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.854131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.853450    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b09ad6fd-125c-4f54-81c5-5862579af346-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.854131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.853517    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.854131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.853723    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.854131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.853922    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.854131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.853926    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.855719 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.855569    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.856138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.855805    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.856210 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.856135    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.856784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.856760    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b09ad6fd-125c-4f54-81c5-5862579af346-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.863340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.863319    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7qc\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-kube-api-access-th7qc\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.863783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.863764    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpj9\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-kube-api-access-wmpj9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.864207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.864175    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:54.864324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.864309    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.961066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.960409    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:54.982679 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:54.972323    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:55.554810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:55.554782    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"]
Oct 31 01:53:55.555617 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:55.555590    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09ad6fd_125c_4f54_81c5_5862579af346.slice/crio-612779b000360772e4c138baf23541ad8fbea0a2136ffe37453bcca3aa35a65d WatchSource:0}: Error finding container 612779b000360772e4c138baf23541ad8fbea0a2136ffe37453bcca3aa35a65d: Status 404 returned error can't find the container with id 612779b000360772e4c138baf23541ad8fbea0a2136ffe37453bcca3aa35a65d
Oct 31 01:53:55.558236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:55.558203    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:55.558332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:55.558279    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:55.558332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:55.558324    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:55.566710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:55.566691    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"]
Oct 31 01:53:55.570082 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:53:55.570022    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefe17a5_e46f_424f_82bf_b030fa88e212.slice/crio-c4862286abd3e280993b439507d6a98c601c07d4f63264bb506df89d3fdae057 WatchSource:0}: Error finding container c4862286abd3e280993b439507d6a98c601c07d4f63264bb506df89d3fdae057: Status 404 returned error can't find the container with id c4862286abd3e280993b439507d6a98c601c07d4f63264bb506df89d3fdae057
Oct 31 01:53:55.574491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:55.574456    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:55.574604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:55.574540    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:55.574604 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:55.574580    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:53:56.058218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.058184    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"]
Oct 31 01:53:56.068077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.068007    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-925bac-m49jg"]
Oct 31 01:53:56.460065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.453319    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf27db1d-91ff-4e40-9115-25b718b0a88f" path="/var/lib/kubelet/pods/cf27db1d-91ff-4e40-9115-25b718b0a88f/volumes"
Oct 31 01:53:56.717704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.710963    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" event={"ID":"aefe17a5-e46f-424f-82bf-b030fa88e212","Type":"ContainerStarted","Data":"2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da"}
Oct 31 01:53:56.717704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.711032    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" event={"ID":"aefe17a5-e46f-424f-82bf-b030fa88e212","Type":"ContainerStarted","Data":"c4862286abd3e280993b439507d6a98c601c07d4f63264bb506df89d3fdae057"}
Oct 31 01:53:56.722949 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.720637    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" event={"ID":"b09ad6fd-125c-4f54-81c5-5862579af346","Type":"ContainerStarted","Data":"63dbdad0030647a6d4c9ebece335649da9feac4ef9dd9151927fa92bf3b0953c"}
Oct 31 01:53:56.722949 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.720704    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" event={"ID":"b09ad6fd-125c-4f54-81c5-5862579af346","Type":"ContainerStarted","Data":"612779b000360772e4c138baf23541ad8fbea0a2136ffe37453bcca3aa35a65d"}
Oct 31 01:53:56.798807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.798736    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podStartSLOduration=2.798713696 podStartE2EDuration="2.798713696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:56.754233871 +0000 UTC m=+1994.702882261" watchObservedRunningTime="2026-10-31 01:53:56.798713696 +0000 UTC m=+1994.747362089"
Oct 31 01:53:56.847300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.847239    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podStartSLOduration=2.84721706 podStartE2EDuration="2.84721706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:56.788874211 +0000 UTC m=+1994.737522603" watchObservedRunningTime="2026-10-31 01:53:56.84721706 +0000 UTC m=+1994.795865453"
Oct 31 01:53:56.900068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.888505    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"]
Oct 31 01:53:56.934144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.907619    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:56.970120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.965800    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:53:56.970120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.965926    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:53:56.970120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:56.965964    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:53:57.007064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.001962    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:53:57.007064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.002119    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:53:57.007064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.002174    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.025834    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"]
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031442    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecaf1d92-4f22-44db-8499-fd49a0348902-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031497    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031531    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgsz\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-kube-api-access-sxgsz\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031576    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031706    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031738    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031772    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031857    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.037151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.031960    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.093118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.088599    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.154975    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecaf1d92-4f22-44db-8499-fd49a0348902-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.155102    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.155150    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgsz\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-kube-api-access-sxgsz\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.155197    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.155352    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.155401    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.155502    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.155622    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.156518    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.157206    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecaf1d92-4f22-44db-8499-fd49a0348902-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.158450    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.168066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.159143    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.264120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.249535    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.264120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.250448    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgsz\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-kube-api-access-sxgsz\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.298067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.285006    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.298067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.285543    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.298067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.287944    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.301453 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.301393    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.406167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.406134    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:53:57.962465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.962429    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:53:57.962715 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:57.962690    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:53:58.159066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:58.071883    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:53:58.159066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:58.071948    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:53:58.232118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:58.231727    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.357068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:58.356137    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.702064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:58.696152    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.863127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:58.710398    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:53:58.863127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:58.813367    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.917315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:58.917247    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.059301 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.059259    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:53:59.119566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.119467    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:53:59.119917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.062433    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 31 01:53:59.261166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.256229    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.384117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.382714    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:53:59.394097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.385462    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.460498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.408645    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:53:59.460818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.460776    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:53:59.493411 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:53:59.493375    5293 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.326s"
Oct 31 01:53:59.509068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.506307    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:53:59.510099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.510061    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.517164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.517118    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:53:59.543529 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.543498    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:53:59.571070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.557276    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"]
Oct 31 01:53:59.571070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.557763    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:53:59.571070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.558142    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:53:59.715065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.714819    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd" podUID="a1e82e90-a1d5-453d-ac98-c37d03673798" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.719067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.716732    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.719067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.716804    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:53:59.741204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.741150    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.857066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.832238    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.887071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.867630    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:53:59.937282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.937241    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:53:59.937646 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.937614    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:53:59.996906 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.996810    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:53:59.997200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:53:59.997158    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:00.025920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.025677    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:00.026309 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.026287    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:54:00.064227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.064180    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v" event={"ID":"ecaf1d92-4f22-44db-8499-fd49a0348902","Type":"ContainerStarted","Data":"900f2fa9de145c3ad7793e98a278bce5ead1c9893b108e507951897225e50027"}
Oct 31 01:54:00.094807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.094757    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:54:00.095272 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.095242    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:54:00.095507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.095471    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:54:00.369274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.345084    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:54:00.390062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.386127    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:54:00.390062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.386387    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:54:00.416540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.390367    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 31 01:54:00.448204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.448169    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:54:00.556063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.543283    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:00.590868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.590826    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:00.618135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.592761    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:00.622953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.622876    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:00.649494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.649455    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 01:54:00.717314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.717261    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.094078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.980710    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:01.094078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:00.980854    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:01.094078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.034003    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:54:01.094078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.034084    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:54:01.094078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.034119    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:54:01.094078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.034149    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:54:01.094078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.055408    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.094778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.094737    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:01.097358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.097313    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.098870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.098832    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.124833 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.124802    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:01.125145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.125098    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:01.180066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.164626    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:01.180376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.180335    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:01.590245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.590199    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.603154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.603113    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.621531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.621489    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.625165 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.625114    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:01.649148 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.590263    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:01.726649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.704932    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.726649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.705334    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:54:01.740169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:01.740118    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:02.057131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.055524    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:02.106092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.096211    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:02.106092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.096274    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:02.146661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.140890    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:02.146661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.140952    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:02.146661 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.142188    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:54:02.155781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.155744    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:54:02.369318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.369205    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:02.713062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.712701    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:02.735069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.731811    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:54:02.747075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.746677    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:02.778093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.709792    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:02.884067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.882081    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:54:02.884067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.883440    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:02.894071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.884876    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:54:02.894071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.885547    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:54:02.959167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.952302    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:54:03.094727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.966344    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:03.094727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.966414    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:03.094727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.997752    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:03.094727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:02.997819    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:03.094727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.054139    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 31 01:54:03.094727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.060207    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:03.221629 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.221577    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:54:03.381809 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.381723    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"]
Oct 31 01:54:03.519107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.506435    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:54:03.680523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.680482    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:03.727318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.727278    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v" event={"ID":"ecaf1d92-4f22-44db-8499-fd49a0348902","Type":"ContainerStarted","Data":"66c7732e51f7bc858e6f734241582c2673a2b5f0e3b7b2758a18fd949d495398"}
Oct 31 01:54:03.727882 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.727861    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"]
Oct 31 01:54:03.728067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.728032    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:54:03.743187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.731153    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:03.749125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.748885    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:03.751155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.751136    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:03.751350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.751334    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:03.761063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.751536    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:54:03.762370 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.752299    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:03.764150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.764125    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:54:03.783784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.783690    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:03.899294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.865738    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:03.924495 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.924448    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cck8g\" (UniqueName: \"kubernetes.io/projected/80979ac2-369d-45db-92fe-8039d0ed4787-kube-api-access-cck8g\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:03.950120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.924915    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:03.950573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.909719    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:54:03.984069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.909664    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:03.985145 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:03.909700    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:54:04.086115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.082133    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:04.155071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.145354    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:04.155071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.145420    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:04.155071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.145886    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:04.155071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.145929    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:04.242389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.239780    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:04.275066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.270296    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cck8g\" (UniqueName: \"kubernetes.io/projected/80979ac2-369d-45db-92fe-8039d0ed4787-kube-api-access-cck8g\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:04.275066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.270459    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:04.275066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.270703    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:04.298267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.282434    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:04.403069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.398526    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:04.428064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.412126    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:54:04.496074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.495594    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v" podStartSLOduration=8.495571911 podStartE2EDuration="8.495571911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:03.655883408 +0000 UTC m=+2001.604531802" watchObservedRunningTime="2026-10-31 01:54:04.495571911 +0000 UTC m=+2002.444220305"
Oct 31 01:54:04.512806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.512770    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cck8g\" (UniqueName: \"kubernetes.io/projected/80979ac2-369d-45db-92fe-8039d0ed4787-kube-api-access-cck8g\") pod \"maas-api-revoke-sub-7143d18b-hb4hg\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:04.523469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.515148    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:54:04.692240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.662052    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:54:04.692240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.673210    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:04.692240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.691674    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:04.696679 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.696638    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:04.697601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.697567    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:04.714238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.714201    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:54:04.749241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.747713    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:04.777087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.768913    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:04.814062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.785123    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:04.969066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:04.957544    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:54:05.004680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.004581    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:05.014711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.009584    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:05.014711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.009638    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:05.014711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.010475    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:05.014711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.010514    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:05.083356 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.082454    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:05.209538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.209218    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:05.250339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.250280    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:05.520067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.519092    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"]
Oct 31 01:54:05.520067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.519362    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy" containerID="cri-o://8b13a438d6e1627aee08b44e671e0c22b184652f0d9037b53759ba1e84de53ca" gracePeriod=30
Oct 31 01:54:05.526063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.524228    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:05.752872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.651260    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:54:05.752872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.714117    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:54:05.752872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.716960    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:05.752872 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.737133    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:54:05.816271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.765324    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:54:05.816271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.765402    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:54:05.816271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.794054    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:05.816271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:05.794454    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:54:06.080785 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.070349    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.093122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.092359    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.187612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.187569    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:54:06.220067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.201522    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:06.220067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.201617    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:06.293592 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.291795    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.571948 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.571912    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:06.572873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.572818    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:06.575149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.575120    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.651137 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.637762    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.672113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.670371    5293 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:54:06.672113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.670438    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:54:06.718102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.716408    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.791138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.737110    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.923254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.742176    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.935433 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.925380    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:06.969066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:06.925423    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:54:07.049623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.044913    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:07.049623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.044995    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:07.049623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.047653    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:07.049623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.047736    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:07.076964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.076929    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:54:07.106070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.077581    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:07.124954 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.124909    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:07.222120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.206189    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:07.372491 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.315265    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:54:07.571067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.547959    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg" event={"ID":"80979ac2-369d-45db-92fe-8039d0ed4787","Type":"ContainerStarted","Data":"04cfacfef5d35d595537ae96d884c6863b4203ada48f95725c037a7e3eee7ed9"}
Oct 31 01:54:07.613137 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.578027    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"]
Oct 31 01:54:07.645094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.619215    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:07.645517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.645479    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:07.695069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.675833    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:54:07.739226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.739182    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:07.950065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.937123    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:54:07.950065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.937841    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:07.984402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.981449    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:07.984402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:07.983357    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:54:08.002063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.002009    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:08.035187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.035134    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:08.040064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.004420    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:08.053066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.040310    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:08.053342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.050694    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:08.113322 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.113271    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:08.279123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.250343    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:08.314065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.307925    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:54:08.355287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.353786    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:54:08.398195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.397297    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:54:08.608123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.587775    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:54:08.651070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.619409    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:54:08.651070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.649703    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:54:08.651070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.650922    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:54:08.673209 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.668911    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:54:08.677870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.677806    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:54:08.686167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.679146    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:54:08.686167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.679307    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:08.723068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.722980    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg" event={"ID":"80979ac2-369d-45db-92fe-8039d0ed4787","Type":"ContainerStarted","Data":"106867af56182e7d11888b128e5f30389d2d25f9263bd6877963dbb1d31f0a37"}
Oct 31 01:54:08.747279 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.746958    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:54:08.965302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.965263    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 01:54:08.965902 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.965853    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 01:54:08.976181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.976150    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 31 01:54:08.976355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:08.976199    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 31 01:54:09.753187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:09.743654    5293 generic.go:356] "Generic (PLEG): container finished" podID="80979ac2-369d-45db-92fe-8039d0ed4787" containerID="106867af56182e7d11888b128e5f30389d2d25f9263bd6877963dbb1d31f0a37" exitCode=0
Oct 31 01:54:09.753187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:09.743797    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg" event={"ID":"80979ac2-369d-45db-92fe-8039d0ed4787","Type":"ContainerDied","Data":"106867af56182e7d11888b128e5f30389d2d25f9263bd6877963dbb1d31f0a37"}
Oct 31 01:54:09.966790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:09.963867    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:54:09.976715 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:09.976476    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:54:10.060117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.047005    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:54:10.472607 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.472579    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:54:10.777873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.776340    5293 generic.go:356] "Generic (PLEG): container finished" podID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerID="8b13a438d6e1627aee08b44e671e0c22b184652f0d9037b53759ba1e84de53ca" exitCode=0
Oct 31 01:54:10.777873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.776410    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" event={"ID":"a1c3959c-9fd5-4a05-b004-a27bab56e7a5","Type":"ContainerDied","Data":"8b13a438d6e1627aee08b44e671e0c22b184652f0d9037b53759ba1e84de53ca"}
Oct 31 01:54:10.777873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.776507    5293 scope.go:122] "RemoveContainer" containerID="9a2f37bc55d219e810735f18858dd17a05c378eca95482c3b29686ec9f48cd81"
Oct 31 01:54:10.778957 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.778628    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 01:54:10.779520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.779499    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 01:54:10.907310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.888568    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:54:10.970393 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.970357    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istiod-ca-cert\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.970941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.970834    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istiod-ca-cert" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:10.971010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.970911    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-podinfo\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.972424    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-data\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.972802    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-data" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.973105    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-kube-api-access-fs4n9\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-kube-api-access-fs4n9\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.973605    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-socket\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.973912    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-socket" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.973988    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-credential-socket\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.974233    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-credential-socket" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.974333    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-token\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.974720    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-envoy\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.975113    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-certs\") pod \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\" (UID: \"a1c3959c-9fd5-4a05-b004-a27bab56e7a5\") "
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.975398    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-certs" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.976057    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.976079    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.976093    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.976108    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.976189    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:10.978601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.978150    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-kube-api-access-fs4n9" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "kube-api-access-fs4n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:10.981890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.980622    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-token" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:10.981890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.980837    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-envoy" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:10.981890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:10.980955    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-podinfo" pod "a1c3959c-9fd5-4a05-b004-a27bab56e7a5" (UID: "a1c3959c-9fd5-4a05-b004-a27bab56e7a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:11.045494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.045463    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:11.083064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.078433    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs4n9\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-kube-api-access-fs4n9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:11.083064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.078469    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:11.083064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.078486    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:11.083064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.078497    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3959c-9fd5-4a05-b004-a27bab56e7a5-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:11.294161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.209497    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-xks-ca\") pod \"80979ac2-369d-45db-92fe-8039d0ed4787\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") "
Oct 31 01:54:11.294161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.209562    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80979ac2-369d-45db-92fe-8039d0ed4787-kube-api-access-cck8g\" (UniqueName: \"kubernetes.io/projected/80979ac2-369d-45db-92fe-8039d0ed4787-kube-api-access-cck8g\") pod \"80979ac2-369d-45db-92fe-8039d0ed4787\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") "
Oct 31 01:54:11.294161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.209805    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-openshift-ca\") pod \"80979ac2-369d-45db-92fe-8039d0ed4787\" (UID: \"80979ac2-369d-45db-92fe-8039d0ed4787\") "
Oct 31 01:54:11.294161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.233674    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-xks-ca" pod "80979ac2-369d-45db-92fe-8039d0ed4787" (UID: "80979ac2-369d-45db-92fe-8039d0ed4787"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:11.294161 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.240381    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80979ac2-369d-45db-92fe-8039d0ed4787-kube-api-access-cck8g" pod "80979ac2-369d-45db-92fe-8039d0ed4787" (UID: "80979ac2-369d-45db-92fe-8039d0ed4787"). InnerVolumeSpecName "kube-api-access-cck8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:11.296755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.296611    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:11.305193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.305145    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cck8g\" (UniqueName: \"kubernetes.io/projected/80979ac2-369d-45db-92fe-8039d0ed4787-kube-api-access-cck8g\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:11.330525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.307295    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-openshift-ca" pod "80979ac2-369d-45db-92fe-8039d0ed4787" (UID: "80979ac2-369d-45db-92fe-8039d0ed4787"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:11.407537 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.407465    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80979ac2-369d-45db-92fe-8039d0ed4787-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:11.972143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.947399    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"
Oct 31 01:54:11.972143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.959867    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg" event={"ID":"80979ac2-369d-45db-92fe-8039d0ed4787","Type":"ContainerDied","Data":"04cfacfef5d35d595537ae96d884c6863b4203ada48f95725c037a7e3eee7ed9"}
Oct 31 01:54:11.972143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.959940    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04cfacfef5d35d595537ae96d884c6863b4203ada48f95725c037a7e3eee7ed9"
Oct 31 01:54:11.999627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:11.997433    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"
Oct 31 01:54:12.020392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:12.007696    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22" event={"ID":"a1c3959c-9fd5-4a05-b004-a27bab56e7a5","Type":"ContainerDied","Data":"ddf43563679b7953cce8e21bee1a851ec18e2df6e29d8f7d97ee8e7932aa0da4"}
Oct 31 01:54:12.020392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:12.007765    5293 scope.go:122] "RemoveContainer" containerID="8b13a438d6e1627aee08b44e671e0c22b184652f0d9037b53759ba1e84de53ca"
Oct 31 01:54:12.028594 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:54:12.027155    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:54:12.054367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:12.054332    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"]
Oct 31 01:54:12.058350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:12.058325    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-n9p22"]
Oct 31 01:54:12.213094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:12.210576    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" path="/var/lib/kubelet/pods/a1c3959c-9fd5-4a05-b004-a27bab56e7a5/volumes"
Oct 31 01:54:14.307347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:14.307269    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"]
Oct 31 01:54:14.322132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:14.322101    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"]
Oct 31 01:54:15.042131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.042096    5293 generic.go:356] "Generic (PLEG): container finished" podID="493b0f59-1286-4e4c-a47e-871418369fd9" containerID="93c574bba911cde51456888ecfd597691ad14e0cb5e178fd68713ae896e86112" exitCode=0
Oct 31 01:54:15.042333 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.042189    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" event={"ID":"493b0f59-1286-4e4c-a47e-871418369fd9","Type":"ContainerDied","Data":"93c574bba911cde51456888ecfd597691ad14e0cb5e178fd68713ae896e86112"}
Oct 31 01:54:15.042680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.042656    5293 scope.go:122] "RemoveContainer" containerID="93c574bba911cde51456888ecfd597691ad14e0cb5e178fd68713ae896e86112"
Oct 31 01:54:15.044533 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.044490    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:54:15.044691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.044587    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:54:15.044691 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.044658    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:54:15.960797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.960763    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"]
Oct 31 01:54:15.961452 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.961431    5293 cpu_manager.go:414] "containerMap: removing container" podUID="80979ac2-369d-45db-92fe-8039d0ed4787" containerName="revoke-keys"
Oct 31 01:54:15.961452 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.961451    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80979ac2-369d-45db-92fe-8039d0ed4787" containerName="revoke-keys"
Oct 31 01:54:15.961611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.961467    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy"
Oct 31 01:54:15.961611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.961472    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy"
Oct 31 01:54:15.961686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.961612    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="80979ac2-369d-45db-92fe-8039d0ed4787" containerName="revoke-keys"
Oct 31 01:54:15.961686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.961623    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy"
Oct 31 01:54:15.965287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.965271    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:15.975726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.975705    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:15.975843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.975772    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:54:15.983881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:15.983855    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"]
Oct 31 01:54:16.053966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.053928    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" event={"ID":"493b0f59-1286-4e4c-a47e-871418369fd9","Type":"ContainerStarted","Data":"b02874b823a70910ae0c34e85aed6534998a9f464c499357f9636ea315c25d78"}
Oct 31 01:54:16.059315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.059288    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:16.059475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.059414    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:16.077926 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.077903    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.078098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.078077    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lcb\" (UniqueName: \"kubernetes.io/projected/b7dd4e85-4351-45c2-bc7f-84f0c383f916-kube-api-access-h2lcb\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.078224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.078208    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.179187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.179158    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lcb\" (UniqueName: \"kubernetes.io/projected/b7dd4e85-4351-45c2-bc7f-84f0c383f916-kube-api-access-h2lcb\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.179372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.179303    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.179490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.179471    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.180183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.180156    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.181817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.181797    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.194858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.194831    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lcb\" (UniqueName: \"kubernetes.io/projected/b7dd4e85-4351-45c2-bc7f-84f0c383f916-kube-api-access-h2lcb\") pod \"maas-api-revoke-sub-348ecd34-x26dq\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.280566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.280487    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:16.492868 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.492839    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:54:16.494904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.494514    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:54:16.494904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.494572    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:54:16.993935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:16.992407    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"]
Oct 31 01:54:17.075689 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:17.075615    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq" event={"ID":"b7dd4e85-4351-45c2-bc7f-84f0c383f916","Type":"ContainerStarted","Data":"9c475163eb87eb0ddb01449bb382fe134f7309fd8b5bb65f5e63acce5de75fc8"}
Oct 31 01:54:17.493569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:17.493534    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:54:17.493827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:17.493786    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:54:18.085532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:18.085487    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq" event={"ID":"b7dd4e85-4351-45c2-bc7f-84f0c383f916","Type":"ContainerStarted","Data":"0460fd6dffa3dce30168355a9b02afc8d73078d50486c425c6c6c0052a292643"}
Oct 31 01:54:18.131984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:18.131926    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq" podStartSLOduration=3.13191096 podStartE2EDuration="3.13191096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:18.109256151 +0000 UTC m=+2016.057904543" watchObservedRunningTime="2026-10-31 01:54:18.13191096 +0000 UTC m=+2016.080559353"
Oct 31 01:54:18.492828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:18.492789    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:54:18.493024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:18.492847    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:54:19.171066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.153032    5293 generic.go:356] "Generic (PLEG): container finished" podID="b7dd4e85-4351-45c2-bc7f-84f0c383f916" containerID="0460fd6dffa3dce30168355a9b02afc8d73078d50486c425c6c6c0052a292643" exitCode=0
Oct 31 01:54:19.171066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.153252    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq" event={"ID":"b7dd4e85-4351-45c2-bc7f-84f0c383f916","Type":"ContainerDied","Data":"0460fd6dffa3dce30168355a9b02afc8d73078d50486c425c6c6c0052a292643"}
Oct 31 01:54:19.319062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.318188    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk" podUID="d44079c0-9086-441e-8711-1b85236abb2b" containerName="payload-processing" containerID="cri-o://6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e" gracePeriod=25
Oct 31 01:54:19.332265 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.330023    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf" podUID="8f8ccd6c-d000-4828-8d9a-7d8765111534" containerName="payload-pre-processing" containerID="cri-o://a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d" gracePeriod=25
Oct 31 01:54:19.492867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.492831    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:54:19.493088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.492890    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:54:19.840250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.839853    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:54:19.857312 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.857288    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:54:19.937309 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.937198    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44079c0-9086-441e-8711-1b85236abb2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44079c0-9086-441e-8711-1b85236abb2b-plugins-config-volume\") pod \"d44079c0-9086-441e-8711-1b85236abb2b\" (UID: \"d44079c0-9086-441e-8711-1b85236abb2b\") "
Oct 31 01:54:19.937500 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.937322    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8ccd6c-d000-4828-8d9a-7d8765111534-kube-api-access-tjrtv\" (UniqueName: \"kubernetes.io/projected/8f8ccd6c-d000-4828-8d9a-7d8765111534-kube-api-access-tjrtv\") pod \"8f8ccd6c-d000-4828-8d9a-7d8765111534\" (UID: \"8f8ccd6c-d000-4828-8d9a-7d8765111534\") "
Oct 31 01:54:19.937567 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.937493    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44079c0-9086-441e-8711-1b85236abb2b-kube-api-access-7gp26\" (UniqueName: \"kubernetes.io/projected/d44079c0-9086-441e-8711-1b85236abb2b-kube-api-access-7gp26\") pod \"d44079c0-9086-441e-8711-1b85236abb2b\" (UID: \"d44079c0-9086-441e-8711-1b85236abb2b\") "
Oct 31 01:54:19.937625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.937587    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8ccd6c-d000-4828-8d9a-7d8765111534-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8ccd6c-d000-4828-8d9a-7d8765111534-plugins-config-volume\") pod \"8f8ccd6c-d000-4828-8d9a-7d8765111534\" (UID: \"8f8ccd6c-d000-4828-8d9a-7d8765111534\") "
Oct 31 01:54:19.942059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.940006    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44079c0-9086-441e-8711-1b85236abb2b-plugins-config-volume" pod "d44079c0-9086-441e-8711-1b85236abb2b" (UID: "d44079c0-9086-441e-8711-1b85236abb2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:19.942059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.940189    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8ccd6c-d000-4828-8d9a-7d8765111534-plugins-config-volume" pod "8f8ccd6c-d000-4828-8d9a-7d8765111534" (UID: "8f8ccd6c-d000-4828-8d9a-7d8765111534"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:19.942378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.942331    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8ccd6c-d000-4828-8d9a-7d8765111534-kube-api-access-tjrtv" pod "8f8ccd6c-d000-4828-8d9a-7d8765111534" (UID: "8f8ccd6c-d000-4828-8d9a-7d8765111534"). InnerVolumeSpecName "kube-api-access-tjrtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:19.942935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:19.942908    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44079c0-9086-441e-8711-1b85236abb2b-kube-api-access-7gp26" pod "d44079c0-9086-441e-8711-1b85236abb2b" (UID: "d44079c0-9086-441e-8711-1b85236abb2b"). InnerVolumeSpecName "kube-api-access-7gp26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:20.039975 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.039847    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44079c0-9086-441e-8711-1b85236abb2b-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:20.039975 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.039878    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrtv\" (UniqueName: \"kubernetes.io/projected/8f8ccd6c-d000-4828-8d9a-7d8765111534-kube-api-access-tjrtv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:20.039975 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.039887    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gp26\" (UniqueName: \"kubernetes.io/projected/d44079c0-9086-441e-8711-1b85236abb2b-kube-api-access-7gp26\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:20.039975 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.039897    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8ccd6c-d000-4828-8d9a-7d8765111534-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:20.163363 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.163327    5293 generic.go:356] "Generic (PLEG): container finished" podID="d44079c0-9086-441e-8711-1b85236abb2b" containerID="6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e" exitCode=0
Oct 31 01:54:20.163563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.163374    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"
Oct 31 01:54:20.163563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.163404    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk" event={"ID":"d44079c0-9086-441e-8711-1b85236abb2b","Type":"ContainerDied","Data":"6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e"}
Oct 31 01:54:20.163563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.163453    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk" event={"ID":"d44079c0-9086-441e-8711-1b85236abb2b","Type":"ContainerDied","Data":"0c27d4c097faee50ad7e3016891c3fdb437eaed5b302e1dfbe4e0a4d4610ed3e"}
Oct 31 01:54:20.163563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.163475    5293 scope.go:122] "RemoveContainer" containerID="6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e"
Oct 31 01:54:20.168344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.168256    5293 generic.go:356] "Generic (PLEG): container finished" podID="8f8ccd6c-d000-4828-8d9a-7d8765111534" containerID="a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d" exitCode=0
Oct 31 01:54:20.168465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.168376    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"
Oct 31 01:54:20.176795 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.176770    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf" event={"ID":"8f8ccd6c-d000-4828-8d9a-7d8765111534","Type":"ContainerDied","Data":"a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d"}
Oct 31 01:54:20.177232 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.176815    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf" event={"ID":"8f8ccd6c-d000-4828-8d9a-7d8765111534","Type":"ContainerDied","Data":"b93718fb50c9ee9249af2afbdaa131f17c0569ea484c72bd40eb6896c27700c1"}
Oct 31 01:54:20.180150 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.180128    5293 scope.go:122] "RemoveContainer" containerID="6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e"
Oct 31 01:54:20.180644 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:54:20.180498    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e\": container with ID starting with 6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e not found: ID does not exist" containerID="6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e"
Oct 31 01:54:20.180644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.180536    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e"} err="failed to get container status \"6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e\": rpc error: code = NotFound desc = could not find container \"6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e\": container with ID starting with 6a76b467294f64ae23480a2b488d4f689dbfa01f8a7cf86e04bb4868f25dbf0e not found: ID does not exist"
Oct 31 01:54:20.180644 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.180562    5293 scope.go:122] "RemoveContainer" containerID="a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d"
Oct 31 01:54:20.194612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.194588    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"]
Oct 31 01:54:20.229067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.227237    5293 scope.go:122] "RemoveContainer" containerID="a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d"
Oct 31 01:54:20.229067 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:54:20.228416    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d\": container with ID starting with a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d not found: ID does not exist" containerID="a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d"
Oct 31 01:54:20.229067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.228456    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d"} err="failed to get container status \"a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d\": rpc error: code = NotFound desc = could not find container \"a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d\": container with ID starting with a06cdcd8d685c9208f4db89733cfe6c322c42e58abd1f8cd2ae3530d3f69b18d not found: ID does not exist"
Oct 31 01:54:20.234296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.234176    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f65e1-57d7b54575-2mxkk"]
Oct 31 01:54:20.314120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.312108    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"]
Oct 31 01:54:20.461848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.461808    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f65e1-5dc87f69f5-84wdf"]
Oct 31 01:54:20.493480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.492981    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:54:20.493480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.493070    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:54:20.557285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.557232    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"]
Oct 31 01:54:20.558106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.558073    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy" containerID="cri-o://debe4a6c1264ca2755db3ac5233026de6ab150c20f3a750c7aa41da9ce76bf89" gracePeriod=30
Oct 31 01:54:20.561611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.561558    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"]
Oct 31 01:54:20.561801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.561748    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v" podUID="ecaf1d92-4f22-44db-8499-fd49a0348902" containerName="istio-proxy" containerID="cri-o://66c7732e51f7bc858e6f734241582c2673a2b5f0e3b7b2758a18fd949d495398" gracePeriod=30
Oct 31 01:54:20.740335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.740282    5293 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 01:54:20.741136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.741094    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v" podUID="ecaf1d92-4f22-44db-8499-fd49a0348902" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:54:20.936107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:20.933749    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:21.124144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.117607    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:54:21.133514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.133336    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7dd4e85-4351-45c2-bc7f-84f0c383f916-kube-api-access-h2lcb\" (UniqueName: \"kubernetes.io/projected/b7dd4e85-4351-45c2-bc7f-84f0c383f916-kube-api-access-h2lcb\") pod \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") "
Oct 31 01:54:21.133714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.133522    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-xks-ca\") pod \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") "
Oct 31 01:54:21.133714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.133602    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-openshift-ca\") pod \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\" (UID: \"b7dd4e85-4351-45c2-bc7f-84f0c383f916\") "
Oct 31 01:54:21.137427 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.137390    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-openshift-ca" pod "b7dd4e85-4351-45c2-bc7f-84f0c383f916" (UID: "b7dd4e85-4351-45c2-bc7f-84f0c383f916"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:21.138254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.138224    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:21.139867 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.139846    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-xks-ca" pod "b7dd4e85-4351-45c2-bc7f-84f0c383f916" (UID: "b7dd4e85-4351-45c2-bc7f-84f0c383f916"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:21.165276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.165246    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7dd4e85-4351-45c2-bc7f-84f0c383f916-kube-api-access-h2lcb" pod "b7dd4e85-4351-45c2-bc7f-84f0c383f916" (UID: "b7dd4e85-4351-45c2-bc7f-84f0c383f916"). InnerVolumeSpecName "kube-api-access-h2lcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:21.181057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.180944    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"
Oct 31 01:54:21.181057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.180974    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq" event={"ID":"b7dd4e85-4351-45c2-bc7f-84f0c383f916","Type":"ContainerDied","Data":"9c475163eb87eb0ddb01449bb382fe134f7309fd8b5bb65f5e63acce5de75fc8"}
Oct 31 01:54:21.181057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.181029    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c475163eb87eb0ddb01449bb382fe134f7309fd8b5bb65f5e63acce5de75fc8"
Oct 31 01:54:21.240906 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.240871    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2lcb\" (UniqueName: \"kubernetes.io/projected/b7dd4e85-4351-45c2-bc7f-84f0c383f916-kube-api-access-h2lcb\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:21.240906 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.240902    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7dd4e85-4351-45c2-bc7f-84f0c383f916-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:21.492842 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.492808    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:54:21.493025 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:21.492874    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:54:22.176068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:22.176017    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8ccd6c-d000-4828-8d9a-7d8765111534" path="/var/lib/kubelet/pods/8f8ccd6c-d000-4828-8d9a-7d8765111534/volumes"
Oct 31 01:54:22.176525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:22.176510    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44079c0-9086-441e-8711-1b85236abb2b" path="/var/lib/kubelet/pods/d44079c0-9086-441e-8711-1b85236abb2b/volumes"
Oct 31 01:54:22.497419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:22.497345    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:54:22.498642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:22.498618    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.981303    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"]
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982323    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8f8ccd6c-d000-4828-8d9a-7d8765111534" containerName="payload-pre-processing"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982346    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8ccd6c-d000-4828-8d9a-7d8765111534" containerName="payload-pre-processing"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982385    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982393    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982422    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b7dd4e85-4351-45c2-bc7f-84f0c383f916" containerName="revoke-keys"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982429    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7dd4e85-4351-45c2-bc7f-84f0c383f916" containerName="revoke-keys"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982447    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d44079c0-9086-441e-8711-1b85236abb2b" containerName="payload-processing"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982455    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44079c0-9086-441e-8711-1b85236abb2b" containerName="payload-processing"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982620    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7dd4e85-4351-45c2-bc7f-84f0c383f916" containerName="revoke-keys"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982636    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44079c0-9086-441e-8711-1b85236abb2b" containerName="payload-processing"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982654    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8ccd6c-d000-4828-8d9a-7d8765111534" containerName="payload-pre-processing"
Oct 31 01:54:25.000065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:24.982668    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c3959c-9fd5-4a05-b004-a27bab56e7a5" containerName="istio-proxy"
Oct 31 01:54:25.034408 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.034379    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"]
Oct 31 01:54:25.034574 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.034478    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.038384 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.038353    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:25.038538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.038449    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:25.052245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.052147    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:54:25.052245 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.052207    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:25.110395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.110362    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:54:25.150162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.150124    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.150335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.150198    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9565\" (UniqueName: \"kubernetes.io/projected/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-kube-api-access-d9565\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.150475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.150442    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.251800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.251714    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.251979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.251943    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.252071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.251993    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9565\" (UniqueName: \"kubernetes.io/projected/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-kube-api-access-d9565\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.252553 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.252528    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.254615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.254580    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.265896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.265872    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9565\" (UniqueName: \"kubernetes.io/projected/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-kube-api-access-d9565\") pod \"maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.350970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.350937    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:25.753230 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:25.753194    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"]
Oct 31 01:54:26.332267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.313984    5293 generic.go:356] "Generic (PLEG): container finished" podID="ecaf1d92-4f22-44db-8499-fd49a0348902" containerID="66c7732e51f7bc858e6f734241582c2673a2b5f0e3b7b2758a18fd949d495398" exitCode=0
Oct 31 01:54:26.332267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.314616    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v" event={"ID":"ecaf1d92-4f22-44db-8499-fd49a0348902","Type":"ContainerDied","Data":"66c7732e51f7bc858e6f734241582c2673a2b5f0e3b7b2758a18fd949d495398"}
Oct 31 01:54:26.332267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.318833    5293 generic.go:356] "Generic (PLEG): container finished" podID="1b865fc7-6a25-467c-b154-45b71658520c" containerID="debe4a6c1264ca2755db3ac5233026de6ab150c20f3a750c7aa41da9ce76bf89" exitCode=0
Oct 31 01:54:26.332267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.319028    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" event={"ID":"1b865fc7-6a25-467c-b154-45b71658520c","Type":"ContainerDied","Data":"debe4a6c1264ca2755db3ac5233026de6ab150c20f3a750c7aa41da9ce76bf89"}
Oct 31 01:54:26.332267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.319109    5293 scope.go:122] "RemoveContainer" containerID="86c8a4e0628769a4bc09c955e834d9843ab7c45be5753ea4bf6b4c30c27d3ccf"
Oct 31 01:54:26.332267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.331418    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq" event={"ID":"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c","Type":"ContainerStarted","Data":"3778c70718da04ca5f94854b0900c6e06fd7add37ad417c9c7d3c5bb5b3636e5"}
Oct 31 01:54:26.407131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.406892    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:54:26.497090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.496534    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505366    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-socket\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505481    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-kube-api-access-7kwpb\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-kube-api-access-7kwpb\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505528    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-certs\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505566    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b865fc7-6a25-467c-b154-45b71658520c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b865fc7-6a25-467c-b154-45b71658520c-istiod-ca-cert\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505601    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-data\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505668    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b865fc7-6a25-467c-b154-45b71658520c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b865fc7-6a25-467c-b154-45b71658520c-istio-podinfo\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505706    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-credential-socket\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505770    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-istio-token\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-istio-token\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.513129 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.505913    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-envoy\") pod \"1b865fc7-6a25-467c-b154-45b71658520c\" (UID: \"1b865fc7-6a25-467c-b154-45b71658520c\") "
Oct 31 01:54:26.530092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.522491    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b865fc7-6a25-467c-b154-45b71658520c-istiod-ca-cert" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:26.530092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.522722    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-socket" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.546520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.546156    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-data" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.548350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.548308    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-certs" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.548839 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.548801    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-credential-socket" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.561825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.561756    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-envoy" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.562149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.562113    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b865fc7-6a25-467c-b154-45b71658520c-istio-podinfo" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:26.562501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.562469    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-kube-api-access-7kwpb" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "kube-api-access-7kwpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.563947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.563919    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-istio-token" pod "1b865fc7-6a25-467c-b154-45b71658520c" (UID: "1b865fc7-6a25-467c-b154-45b71658520c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.734772 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.734740    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecaf1d92-4f22-44db-8499-fd49a0348902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecaf1d92-4f22-44db-8499-fd49a0348902-istiod-ca-cert\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.736142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.736119    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-socket\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.736774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.736699    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-data\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.744135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.744112    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-envoy\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.753271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.753247    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-certs\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.765317 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.765248    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-kube-api-access-sxgsz\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-kube-api-access-sxgsz\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.766191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.735979    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaf1d92-4f22-44db-8499-fd49a0348902-istiod-ca-cert" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:26.766191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.736590    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-socket" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.766191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.743918    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-data" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.766191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.765156    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-certs" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.791475    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-token\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-token\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.791636    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-credential-socket\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.791674    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-podinfo\") pod \"ecaf1d92-4f22-44db-8499-fd49a0348902\" (UID: \"ecaf1d92-4f22-44db-8499-fd49a0348902\") "
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792393    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b865fc7-6a25-467c-b154-45b71658520c-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792416    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792431    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b865fc7-6a25-467c-b154-45b71658520c-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792444    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792459    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792472    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792486    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792499    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecaf1d92-4f22-44db-8499-fd49a0348902-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792514    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792532    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792551    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792571    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwpb\" (UniqueName: \"kubernetes.io/projected/1b865fc7-6a25-467c-b154-45b71658520c-kube-api-access-7kwpb\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.800609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.792589    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b865fc7-6a25-467c-b154-45b71658520c-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.823172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.814166    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-credential-socket" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.890065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.889916    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-token" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.908686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.891457    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-podinfo" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:26.908686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.898628    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.908686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.898655    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.908686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.898671    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.908686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.905583    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-kube-api-access-sxgsz" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "kube-api-access-sxgsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.908686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:26.908398    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-envoy" pod "ecaf1d92-4f22-44db-8499-fd49a0348902" (UID: "ecaf1d92-4f22-44db-8499-fd49a0348902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:27.003067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.002630    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecaf1d92-4f22-44db-8499-fd49a0348902-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:27.003067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.002689    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgsz\" (UniqueName: \"kubernetes.io/projected/ecaf1d92-4f22-44db-8499-fd49a0348902-kube-api-access-sxgsz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:27.418117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.405451    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v" event={"ID":"ecaf1d92-4f22-44db-8499-fd49a0348902","Type":"ContainerDied","Data":"900f2fa9de145c3ad7793e98a278bce5ead1c9893b108e507951897225e50027"}
Oct 31 01:54:27.418117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.405529    5293 scope.go:122] "RemoveContainer" containerID="66c7732e51f7bc858e6f734241582c2673a2b5f0e3b7b2758a18fd949d495398"
Oct 31 01:54:27.418117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.405675    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"
Oct 31 01:54:27.598665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.549824    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb" event={"ID":"1b865fc7-6a25-467c-b154-45b71658520c","Type":"ContainerDied","Data":"a0244a1bcd071570a125bfa5c0f9d48354f3dc800d297b86fdf2d85af818e1aa"}
Oct 31 01:54:27.598665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.549988    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"
Oct 31 01:54:27.599382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.599305    5293 generic.go:356] "Generic (PLEG): container finished" podID="1e6bfdf7-1e14-4ae2-b453-c5d393e8255c" containerID="0f670f75c85109d48f743bbd85ac4e4f9875b4b4a50d133b4d03f45f1d34960f" exitCode=0
Oct 31 01:54:27.654571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.647467    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq" event={"ID":"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c","Type":"ContainerDied","Data":"0f670f75c85109d48f743bbd85ac4e4f9875b4b4a50d133b4d03f45f1d34960f"}
Oct 31 01:54:27.654571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.647601    5293 scope.go:122] "RemoveContainer" containerID="debe4a6c1264ca2755db3ac5233026de6ab150c20f3a750c7aa41da9ce76bf89"
Oct 31 01:54:27.889656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.889616    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"]
Oct 31 01:54:27.962064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:27.955626    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-pll4v"]
Oct 31 01:54:28.014240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.014199    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"]
Oct 31 01:54:28.034496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.034140    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.036134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.036099    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-q9ztb"]
Oct 31 01:54:28.063186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.062128    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.152123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.151131    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.187206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.187159    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.231672 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.231633    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b865fc7-6a25-467c-b154-45b71658520c" path="/var/lib/kubelet/pods/1b865fc7-6a25-467c-b154-45b71658520c/volumes"
Oct 31 01:54:28.252417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.252386    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecaf1d92-4f22-44db-8499-fd49a0348902" path="/var/lib/kubelet/pods/ecaf1d92-4f22-44db-8499-fd49a0348902/volumes"
Oct 31 01:54:28.366120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.363125    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:54:28.681173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.681130    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.738935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.738385    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.788808 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.788248    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:54:28.806121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.805134    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.806578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.806370    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.892308 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:28.892181    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:54:29.079630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.079585    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:29.080459 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.080433    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:54:29.216566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.175134    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:29.216566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.175557    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:54:29.379914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.378647    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:29.379914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.379201    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:54:29.591412 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.574176    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:54:29.605358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.605294    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:54:29.623106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.611430    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:54:29.623106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.619684    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:29.651170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.650206    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"
Oct 31 01:54:29.651170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.650370    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq" event={"ID":"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c","Type":"ContainerDied","Data":"3778c70718da04ca5f94854b0900c6e06fd7add37ad417c9c7d3c5bb5b3636e5"}
Oct 31 01:54:29.651170 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.650407    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3778c70718da04ca5f94854b0900c6e06fd7add37ad417c9c7d3c5bb5b3636e5"
Oct 31 01:54:29.776865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.776781    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-kube-api-access-d9565\" (UniqueName: \"kubernetes.io/projected/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-kube-api-access-d9565\") pod \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") "
Oct 31 01:54:29.779672 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.779624    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-openshift-ca\") pod \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") "
Oct 31 01:54:29.780877 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.780827    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-openshift-ca" pod "1e6bfdf7-1e14-4ae2-b453-c5d393e8255c" (UID: "1e6bfdf7-1e14-4ae2-b453-c5d393e8255c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:29.781327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.781263    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-xks-ca\") pod \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\" (UID: \"1e6bfdf7-1e14-4ae2-b453-c5d393e8255c\") "
Oct 31 01:54:29.783310 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.783288    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:29.784215 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.784185    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-kube-api-access-d9565" pod "1e6bfdf7-1e14-4ae2-b453-c5d393e8255c" (UID: "1e6bfdf7-1e14-4ae2-b453-c5d393e8255c"). InnerVolumeSpecName "kube-api-access-d9565". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:29.787198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.787174    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-xks-ca" pod "1e6bfdf7-1e14-4ae2-b453-c5d393e8255c" (UID: "1e6bfdf7-1e14-4ae2-b453-c5d393e8255c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:29.886054 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.885962    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:29.886054 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:29.885989    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9565\" (UniqueName: \"kubernetes.io/projected/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c-kube-api-access-d9565\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:32.053060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.053009    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"]
Oct 31 01:54:32.053916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.053893    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy"
Oct 31 01:54:32.054085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.053917    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy"
Oct 31 01:54:32.054085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.053951    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ecaf1d92-4f22-44db-8499-fd49a0348902" containerName="istio-proxy"
Oct 31 01:54:32.054085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.053960    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaf1d92-4f22-44db-8499-fd49a0348902" containerName="istio-proxy"
Oct 31 01:54:32.054085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.054004    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1e6bfdf7-1e14-4ae2-b453-c5d393e8255c" containerName="revoke-keys"
Oct 31 01:54:32.054085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.054014    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6bfdf7-1e14-4ae2-b453-c5d393e8255c" containerName="revoke-keys"
Oct 31 01:54:32.054361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.054202    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaf1d92-4f22-44db-8499-fd49a0348902" containerName="istio-proxy"
Oct 31 01:54:32.054361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.054225    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6bfdf7-1e14-4ae2-b453-c5d393e8255c" containerName="revoke-keys"
Oct 31 01:54:32.054361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.054236    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy"
Oct 31 01:54:32.054361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.054250    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy"
Oct 31 01:54:32.058705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.058686    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.068010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.067926    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:32.068010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.067972    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:32.068010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.067970    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:54:32.068010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.067995    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:32.069963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.069937    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"]
Oct 31 01:54:32.152399 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.152364    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.152654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.152413    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pc7\" (UniqueName: \"kubernetes.io/projected/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-kube-api-access-x5pc7\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.152654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.152495    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.274089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.258386    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.274089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.258451    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pc7\" (UniqueName: \"kubernetes.io/projected/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-kube-api-access-x5pc7\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.274089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.258496    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.274089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.259385    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.299070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.292488    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.379169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.375874    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pc7\" (UniqueName: \"kubernetes.io/projected/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-kube-api-access-x5pc7\") pod \"maas-api-revoke-sub-390dcf31-7dqbf\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.392096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.389316    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:32.827119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:32.826667    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"]
Oct 31 01:54:32.829168 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:54:32.829140    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67da2d4_da3b_4313_bd5e_cdc54c2778d8.slice/crio-66ebec565e42706a5533202bf296b9bf587f05416d498195646d4cf1021ae25c WatchSource:0}: Error finding container 66ebec565e42706a5533202bf296b9bf587f05416d498195646d4cf1021ae25c: Status 404 returned error can't find the container with id 66ebec565e42706a5533202bf296b9bf587f05416d498195646d4cf1021ae25c
Oct 31 01:54:33.729022 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:33.727562    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf" event={"ID":"b67da2d4-da3b-4313-bd5e-cdc54c2778d8","Type":"ContainerStarted","Data":"668d5c07b2b13073bf23ee594701e27d6115b82321c5260b0e2673f879cad185"}
Oct 31 01:54:33.729022 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:33.727708    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf" event={"ID":"b67da2d4-da3b-4313-bd5e-cdc54c2778d8","Type":"ContainerStarted","Data":"66ebec565e42706a5533202bf296b9bf587f05416d498195646d4cf1021ae25c"}
Oct 31 01:54:33.746109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:33.746023    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf" podStartSLOduration=1.74600742 podStartE2EDuration="1.74600742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:33.744663034 +0000 UTC m=+2031.693311430" watchObservedRunningTime="2026-10-31 01:54:33.74600742 +0000 UTC m=+2031.694655815"
Oct 31 01:54:34.737437 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:34.737389    5293 generic.go:356] "Generic (PLEG): container finished" podID="b67da2d4-da3b-4313-bd5e-cdc54c2778d8" containerID="668d5c07b2b13073bf23ee594701e27d6115b82321c5260b0e2673f879cad185" exitCode=0
Oct 31 01:54:34.737944 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:34.737578    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf" event={"ID":"b67da2d4-da3b-4313-bd5e-cdc54c2778d8","Type":"ContainerDied","Data":"668d5c07b2b13073bf23ee594701e27d6115b82321c5260b0e2673f879cad185"}
Oct 31 01:54:35.128110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:35.123164    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"]
Oct 31 01:54:35.142071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:35.140200    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"]
Oct 31 01:54:35.939875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:35.939845    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:36.024963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.024879    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-xks-ca\") pod \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") "
Oct 31 01:54:36.025164 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.024963    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-openshift-ca\") pod \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") "
Oct 31 01:54:36.025231 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.025196    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-kube-api-access-x5pc7\" (UniqueName: \"kubernetes.io/projected/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-kube-api-access-x5pc7\") pod \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\" (UID: \"b67da2d4-da3b-4313-bd5e-cdc54c2778d8\") "
Oct 31 01:54:36.025417 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.025395    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-openshift-ca" pod "b67da2d4-da3b-4313-bd5e-cdc54c2778d8" (UID: "b67da2d4-da3b-4313-bd5e-cdc54c2778d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:36.026276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.026130    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:36.027643 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.027578    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-xks-ca" pod "b67da2d4-da3b-4313-bd5e-cdc54c2778d8" (UID: "b67da2d4-da3b-4313-bd5e-cdc54c2778d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:36.027924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.027894    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-kube-api-access-x5pc7" pod "b67da2d4-da3b-4313-bd5e-cdc54c2778d8" (UID: "b67da2d4-da3b-4313-bd5e-cdc54c2778d8"). InnerVolumeSpecName "kube-api-access-x5pc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:36.146066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.139530    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:36.146395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.146343    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5pc7\" (UniqueName: \"kubernetes.io/projected/b67da2d4-da3b-4313-bd5e-cdc54c2778d8-kube-api-access-x5pc7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:36.777064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.773250    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf" event={"ID":"b67da2d4-da3b-4313-bd5e-cdc54c2778d8","Type":"ContainerDied","Data":"66ebec565e42706a5533202bf296b9bf587f05416d498195646d4cf1021ae25c"}
Oct 31 01:54:36.777064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.773318    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66ebec565e42706a5533202bf296b9bf587f05416d498195646d4cf1021ae25c"
Oct 31 01:54:36.777064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:36.773403    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"
Oct 31 01:54:37.506668 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.506628    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"]
Oct 31 01:54:37.507623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.507599    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy"
Oct 31 01:54:37.507705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.507630    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b865fc7-6a25-467c-b154-45b71658520c" containerName="istio-proxy"
Oct 31 01:54:37.507705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.507674    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b67da2d4-da3b-4313-bd5e-cdc54c2778d8" containerName="revoke-keys"
Oct 31 01:54:37.507705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.507682    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67da2d4-da3b-4313-bd5e-cdc54c2778d8" containerName="revoke-keys"
Oct 31 01:54:37.507862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.507847    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67da2d4-da3b-4313-bd5e-cdc54c2778d8" containerName="revoke-keys"
Oct 31 01:54:37.512018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.511989    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:37.520580 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.520557    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"]
Oct 31 01:54:37.520788 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.520767    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-dbff33-dockercfg-2rjln\""
Oct 31 01:54:37.575122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.575096    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-dbff33\""
Oct 31 01:54:37.680067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.677102    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b698a-1305-45df-ad72-b1cf414d31ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm\" (UID: \"137b698a-1305-45df-ad72-b1cf414d31ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:37.680067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.677205    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpt4v\" (UniqueName: \"kubernetes.io/projected/137b698a-1305-45df-ad72-b1cf414d31ec-kube-api-access-tpt4v\") pod \"payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm\" (UID: \"137b698a-1305-45df-ad72-b1cf414d31ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:37.786128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.785388    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b698a-1305-45df-ad72-b1cf414d31ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm\" (UID: \"137b698a-1305-45df-ad72-b1cf414d31ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:37.786128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.785470    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt4v\" (UniqueName: \"kubernetes.io/projected/137b698a-1305-45df-ad72-b1cf414d31ec-kube-api-access-tpt4v\") pod \"payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm\" (UID: \"137b698a-1305-45df-ad72-b1cf414d31ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:37.789072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.786856    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b698a-1305-45df-ad72-b1cf414d31ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm\" (UID: \"137b698a-1305-45df-ad72-b1cf414d31ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:37.903952 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:37.903918    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt4v\" (UniqueName: \"kubernetes.io/projected/137b698a-1305-45df-ad72-b1cf414d31ec-kube-api-access-tpt4v\") pod \"payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm\" (UID: \"137b698a-1305-45df-ad72-b1cf414d31ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:38.145130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:38.144892    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:38.626193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:38.626170    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"]
Oct 31 01:54:38.628427 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:54:38.628395    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137b698a_1305_45df_ad72_b1cf414d31ec.slice/crio-19fbdaecda9f7a495285f1bb4ee8166d44fb85cbb40b36814a4b4f239b50ab77 WatchSource:0}: Error finding container 19fbdaecda9f7a495285f1bb4ee8166d44fb85cbb40b36814a4b4f239b50ab77: Status 404 returned error can't find the container with id 19fbdaecda9f7a495285f1bb4ee8166d44fb85cbb40b36814a4b4f239b50ab77
Oct 31 01:54:38.814545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:38.814511    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm" event={"ID":"137b698a-1305-45df-ad72-b1cf414d31ec","Type":"ContainerStarted","Data":"a89bc306aa776c79cf37b3ef942cc7fdffe58921b9cea4c91cc12fb4d7b781a7"}
Oct 31 01:54:38.814709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:38.814571    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm" event={"ID":"137b698a-1305-45df-ad72-b1cf414d31ec","Type":"ContainerStarted","Data":"19fbdaecda9f7a495285f1bb4ee8166d44fb85cbb40b36814a4b4f239b50ab77"}
Oct 31 01:54:38.814709 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:38.814594    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:38.845859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:38.845806    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm" podStartSLOduration=1.8457907900000001 podStartE2EDuration="1.84579079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:38.844131625 +0000 UTC m=+2036.792780019" watchObservedRunningTime="2026-10-31 01:54:38.84579079 +0000 UTC m=+2036.794439181"
Oct 31 01:54:40.130358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.130308    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz" podUID="8d5817f1-a01d-4951-a3c3-f0e1717a738f" containerName="payload-processing" containerID="cri-o://d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489" gracePeriod=25
Oct 31 01:54:40.140712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.140658    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw" podUID="8cba5706-9923-4e46-a3bf-60b97650625c" containerName="payload-pre-processing" containerID="cri-o://c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf" gracePeriod=25
Oct 31 01:54:40.481743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.481715    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:54:40.487207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.487185    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5817f1-a01d-4951-a3c3-f0e1717a738f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5817f1-a01d-4951-a3c3-f0e1717a738f-plugins-config-volume\") pod \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\" (UID: \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\") "
Oct 31 01:54:40.487342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.487317    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5817f1-a01d-4951-a3c3-f0e1717a738f-kube-api-access-4b8j4\" (UniqueName: \"kubernetes.io/projected/8d5817f1-a01d-4951-a3c3-f0e1717a738f-kube-api-access-4b8j4\") pod \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\" (UID: \"8d5817f1-a01d-4951-a3c3-f0e1717a738f\") "
Oct 31 01:54:40.487570 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.487547    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5817f1-a01d-4951-a3c3-f0e1717a738f-plugins-config-volume" pod "8d5817f1-a01d-4951-a3c3-f0e1717a738f" (UID: "8d5817f1-a01d-4951-a3c3-f0e1717a738f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:40.487883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.487865    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5817f1-a01d-4951-a3c3-f0e1717a738f-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:40.489468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.489448    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5817f1-a01d-4951-a3c3-f0e1717a738f-kube-api-access-4b8j4" pod "8d5817f1-a01d-4951-a3c3-f0e1717a738f" (UID: "8d5817f1-a01d-4951-a3c3-f0e1717a738f"). InnerVolumeSpecName "kube-api-access-4b8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:40.506341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.506315    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:54:40.597063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.595278    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8j4\" (UniqueName: \"kubernetes.io/projected/8d5817f1-a01d-4951-a3c3-f0e1717a738f-kube-api-access-4b8j4\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:40.714139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.698188    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cba5706-9923-4e46-a3bf-60b97650625c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cba5706-9923-4e46-a3bf-60b97650625c-plugins-config-volume\") pod \"8cba5706-9923-4e46-a3bf-60b97650625c\" (UID: \"8cba5706-9923-4e46-a3bf-60b97650625c\") "
Oct 31 01:54:40.714139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.698282    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cba5706-9923-4e46-a3bf-60b97650625c-kube-api-access-thkn6\" (UniqueName: \"kubernetes.io/projected/8cba5706-9923-4e46-a3bf-60b97650625c-kube-api-access-thkn6\") pod \"8cba5706-9923-4e46-a3bf-60b97650625c\" (UID: \"8cba5706-9923-4e46-a3bf-60b97650625c\") "
Oct 31 01:54:40.738074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.732342    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cba5706-9923-4e46-a3bf-60b97650625c-plugins-config-volume" pod "8cba5706-9923-4e46-a3bf-60b97650625c" (UID: "8cba5706-9923-4e46-a3bf-60b97650625c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:40.738074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.735667    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cba5706-9923-4e46-a3bf-60b97650625c-kube-api-access-thkn6" pod "8cba5706-9923-4e46-a3bf-60b97650625c" (UID: "8cba5706-9923-4e46-a3bf-60b97650625c"). InnerVolumeSpecName "kube-api-access-thkn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:40.832924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.806352    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thkn6\" (UniqueName: \"kubernetes.io/projected/8cba5706-9923-4e46-a3bf-60b97650625c-kube-api-access-thkn6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:40.832924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.806388    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cba5706-9923-4e46-a3bf-60b97650625c-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:40.936370 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.936338    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"]
Oct 31 01:54:40.939342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.939314    5293 generic.go:356] "Generic (PLEG): container finished" podID="8d5817f1-a01d-4951-a3c3-f0e1717a738f" containerID="d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489" exitCode=0
Oct 31 01:54:40.939497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.939387    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz" event={"ID":"8d5817f1-a01d-4951-a3c3-f0e1717a738f","Type":"ContainerDied","Data":"d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489"}
Oct 31 01:54:40.939497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.939427    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz" event={"ID":"8d5817f1-a01d-4951-a3c3-f0e1717a738f","Type":"ContainerDied","Data":"10392b5c635f62ba34f34f3927c6d49d9fbb1b5fcf2eb0ec7602b8118533cded"}
Oct 31 01:54:40.939497 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.939448    5293 scope.go:122] "RemoveContainer" containerID="d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489"
Oct 31 01:54:40.939694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.939586    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"
Oct 31 01:54:40.941647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.941625    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77e2b5-2vrvq"]
Oct 31 01:54:40.945005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.944980    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"
Oct 31 01:54:40.945193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.944879    5293 generic.go:356] "Generic (PLEG): container finished" podID="8cba5706-9923-4e46-a3bf-60b97650625c" containerID="c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf" exitCode=0
Oct 31 01:54:40.945193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.944979    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw" event={"ID":"8cba5706-9923-4e46-a3bf-60b97650625c","Type":"ContainerDied","Data":"c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf"}
Oct 31 01:54:40.945344 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.945193    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw" event={"ID":"8cba5706-9923-4e46-a3bf-60b97650625c","Type":"ContainerDied","Data":"c39ae8464b76b065d150c41832763db078d3d06b9c7e32122a34ef4d817b45ed"}
Oct 31 01:54:40.961373 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.961347    5293 scope.go:122] "RemoveContainer" containerID="d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489"
Oct 31 01:54:40.961765 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:54:40.961740    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489\": container with ID starting with d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489 not found: ID does not exist" containerID="d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489"
Oct 31 01:54:40.961846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.961776    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489"} err="failed to get container status \"d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489\": rpc error: code = NotFound desc = could not find container \"d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489\": container with ID starting with d38087c4e53dd52b42036a26dc517023607f06c063f664054afde87b82b9c489 not found: ID does not exist"
Oct 31 01:54:40.961846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.961798    5293 scope.go:122] "RemoveContainer" containerID="c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf"
Oct 31 01:54:40.975122 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.973224    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"]
Oct 31 01:54:40.975690 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.975669    5293 scope.go:122] "RemoveContainer" containerID="c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf"
Oct 31 01:54:40.976498 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:54:40.976152    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf\": container with ID starting with c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf not found: ID does not exist" containerID="c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf"
Oct 31 01:54:40.976498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.976191    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf"} err="failed to get container status \"c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf\": rpc error: code = NotFound desc = could not find container \"c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf\": container with ID starting with c9c5124457ceaa3e9d12e302e62c3dcda3d3b32a9e4d4fb70287c4ba9010ceaf not found: ID does not exist"
Oct 31 01:54:40.977887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.977858    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77e2b5-649795fff-jtszw"]
Oct 31 01:54:40.986619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.986599    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"]
Oct 31 01:54:40.993247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:40.993226    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77e2b5-6f96567b7c-qdkcz"]
Oct 31 01:54:41.838068 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:54:41.836554    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:54:42.174805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:42.174774    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e6bfdf7-1e14-4ae2-b453-c5d393e8255c" path="/var/lib/kubelet/pods/1e6bfdf7-1e14-4ae2-b453-c5d393e8255c/volumes"
Oct 31 01:54:42.175223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:42.175207    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cba5706-9923-4e46-a3bf-60b97650625c" path="/var/lib/kubelet/pods/8cba5706-9923-4e46-a3bf-60b97650625c/volumes"
Oct 31 01:54:42.175595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:42.175581    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5817f1-a01d-4951-a3c3-f0e1717a738f" path="/var/lib/kubelet/pods/8d5817f1-a01d-4951-a3c3-f0e1717a738f/volumes"
Oct 31 01:54:43.208639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:43.208607    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"]
Oct 31 01:54:43.209468 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:43.209431    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy" containerID="cri-o://f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5" gracePeriod=30
Oct 31 01:54:44.006256 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.006215    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"]
Oct 31 01:54:44.006806 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.006793    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8cba5706-9923-4e46-a3bf-60b97650625c" containerName="payload-pre-processing"
Oct 31 01:54:44.006859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.006808    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cba5706-9923-4e46-a3bf-60b97650625c" containerName="payload-pre-processing"
Oct 31 01:54:44.006859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.006832    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8d5817f1-a01d-4951-a3c3-f0e1717a738f" containerName="payload-processing"
Oct 31 01:54:44.006859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.006837    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5817f1-a01d-4951-a3c3-f0e1717a738f" containerName="payload-processing"
Oct 31 01:54:44.006959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.006940    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cba5706-9923-4e46-a3bf-60b97650625c" containerName="payload-pre-processing"
Oct 31 01:54:44.006959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.006951    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5817f1-a01d-4951-a3c3-f0e1717a738f" containerName="payload-processing"
Oct 31 01:54:44.012030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.012013    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.018940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.018882    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:44.018940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.018926    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:44.018940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.018950    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:44.019225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.019117    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:54:44.022093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.021541    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"]
Oct 31 01:54:44.072697 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.072663    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.072840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.072781    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.072894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.072881    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ldw\" (UniqueName: \"kubernetes.io/projected/d9d6020b-1edc-4c4b-b5e9-848635b65061-kube-api-access-d6ldw\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.174743 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.174709    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ldw\" (UniqueName: \"kubernetes.io/projected/d9d6020b-1edc-4c4b-b5e9-848635b65061-kube-api-access-d6ldw\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.174941 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.174866    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.175022 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.174941    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.175616 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.175596    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.177591 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.177556    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.190772 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.190745    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ldw\" (UniqueName: \"kubernetes.io/projected/d9d6020b-1edc-4c4b-b5e9-848635b65061-kube-api-access-d6ldw\") pod \"maas-api-revoke-sub-1e0007c8-jmpvp\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.332751 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.330746    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:44.896801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.896762    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"]
Oct 31 01:54:44.991489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:44.991454    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp" event={"ID":"d9d6020b-1edc-4c4b-b5e9-848635b65061","Type":"ContainerStarted","Data":"8326928ae34dcfd002f8be0d8cc73559dcaecac5185de3d3446c491984c45614"}
Oct 31 01:54:45.340805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.340771    5293 scope.go:122] "RemoveContainer" containerID="76e10329a5e9e5e249e6e7f6c3e0c57a283aac7a7d3998c7039b6a1151585207"
Oct 31 01:54:45.351866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.351606    5293 scope.go:122] "RemoveContainer" containerID="14ada5cfa422255dbeedfa866d053c75c12c8128d1f553ea643c716f48818af9"
Oct 31 01:54:45.362525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.362502    5293 scope.go:122] "RemoveContainer" containerID="c7a2bff4750647951203289a239911b1d65fb426df41d12a0944b3d2d75e3dea"
Oct 31 01:54:45.458960 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.458932    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"]
Oct 31 01:54:45.464852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.464834    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.470681 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.470658    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.470810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.470752    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qks5\" (UniqueName: \"kubernetes.io/projected/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-kube-api-access-2qks5\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.470897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.470879    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.473640 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.473618    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"]
Oct 31 01:54:45.581258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.580085    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.581258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.580272    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.581258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.580371    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qks5\" (UniqueName: \"kubernetes.io/projected/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-kube-api-access-2qks5\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.584203 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.583829    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.600078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.598862    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qks5\" (UniqueName: \"kubernetes.io/projected/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-kube-api-access-2qks5\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.613066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.606301    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637cedcd-k798b\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:45.788070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:45.783473    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:46.214972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:46.214512    5293 generic.go:356] "Generic (PLEG): container finished" podID="d9d6020b-1edc-4c4b-b5e9-848635b65061" containerID="e43b845a5f1663ed2a44395aa3fdd0cff1e2f5cf130a153eb549ba3ba6695bb1" exitCode=0
Oct 31 01:54:46.218702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:46.218670    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp" event={"ID":"d9d6020b-1edc-4c4b-b5e9-848635b65061","Type":"ContainerDied","Data":"e43b845a5f1663ed2a44395aa3fdd0cff1e2f5cf130a153eb549ba3ba6695bb1"}
Oct 31 01:54:46.370404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:46.370378    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"]
Oct 31 01:54:46.372032 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:54:46.372002    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5bf6ea_fce6_47bb_a77f_37bd64ac7453.slice/crio-db6b16593492f02431560660c7c693fed84a495326b0cc1e9c9057749f15359b WatchSource:0}: Error finding container db6b16593492f02431560660c7c693fed84a495326b0cc1e9c9057749f15359b: Status 404 returned error can't find the container with id db6b16593492f02431560660c7c693fed84a495326b0cc1e9c9057749f15359b
Oct 31 01:54:47.269631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.256391    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b" event={"ID":"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453","Type":"ContainerStarted","Data":"91b69fb67a443a9ef8d69ccef7509d5f5c26edb3b0b6f303195d648aed1c174c"}
Oct 31 01:54:47.269631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.256451    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b" event={"ID":"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453","Type":"ContainerStarted","Data":"db6b16593492f02431560660c7c693fed84a495326b0cc1e9c9057749f15359b"}
Oct 31 01:54:47.284582 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.284516    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b" podStartSLOduration=2.284496223 podStartE2EDuration="2.284496223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:47.283797481 +0000 UTC m=+2045.232445873" watchObservedRunningTime="2026-10-31 01:54:47.284496223 +0000 UTC m=+2045.233144615"
Oct 31 01:54:47.444693 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.444661    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:47.534348 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.534280    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d6020b-1edc-4c4b-b5e9-848635b65061-kube-api-access-d6ldw\" (UniqueName: \"kubernetes.io/projected/d9d6020b-1edc-4c4b-b5e9-848635b65061-kube-api-access-d6ldw\") pod \"d9d6020b-1edc-4c4b-b5e9-848635b65061\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") "
Oct 31 01:54:47.534478 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.534428    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-xks-ca\") pod \"d9d6020b-1edc-4c4b-b5e9-848635b65061\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") "
Oct 31 01:54:47.534538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.534481    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-openshift-ca\") pod \"d9d6020b-1edc-4c4b-b5e9-848635b65061\" (UID: \"d9d6020b-1edc-4c4b-b5e9-848635b65061\") "
Oct 31 01:54:47.534877 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.534850    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-openshift-ca" pod "d9d6020b-1edc-4c4b-b5e9-848635b65061" (UID: "d9d6020b-1edc-4c4b-b5e9-848635b65061"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:47.535166 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.535134    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:47.536479 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.536455    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d6020b-1edc-4c4b-b5e9-848635b65061-kube-api-access-d6ldw" pod "d9d6020b-1edc-4c4b-b5e9-848635b65061" (UID: "d9d6020b-1edc-4c4b-b5e9-848635b65061"). InnerVolumeSpecName "kube-api-access-d6ldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:47.536688 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.536666    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-xks-ca" pod "d9d6020b-1edc-4c4b-b5e9-848635b65061" (UID: "d9d6020b-1edc-4c4b-b5e9-848635b65061"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:47.636077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.636027    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ldw\" (UniqueName: \"kubernetes.io/projected/d9d6020b-1edc-4c4b-b5e9-848635b65061-kube-api-access-d6ldw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:47.636077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:47.636077    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d6020b-1edc-4c4b-b5e9-848635b65061-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.282669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.282631    5293 generic.go:356] "Generic (PLEG): container finished" podID="dd5bf6ea-fce6-47bb-a77f-37bd64ac7453" containerID="91b69fb67a443a9ef8d69ccef7509d5f5c26edb3b0b6f303195d648aed1c174c" exitCode=0
Oct 31 01:54:48.282811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.282731    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b" event={"ID":"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453","Type":"ContainerDied","Data":"91b69fb67a443a9ef8d69ccef7509d5f5c26edb3b0b6f303195d648aed1c174c"}
Oct 31 01:54:48.284402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.284384    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"
Oct 31 01:54:48.284530 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.284423    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp" event={"ID":"d9d6020b-1edc-4c4b-b5e9-848635b65061","Type":"ContainerDied","Data":"8326928ae34dcfd002f8be0d8cc73559dcaecac5185de3d3446c491984c45614"}
Oct 31 01:54:48.284530 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.284470    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8326928ae34dcfd002f8be0d8cc73559dcaecac5185de3d3446c491984c45614"
Oct 31 01:54:48.500375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.499482    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"]
Oct 31 01:54:48.500375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.499729    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy" containerID="cri-o://b02874b823a70910ae0c34e85aed6534998a9f464c499357f9636ea315c25d78" gracePeriod=30
Oct 31 01:54:48.565254 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.565229    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:54:48.609234 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.609202    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"]
Oct 31 01:54:48.609859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.609840    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d9d6020b-1edc-4c4b-b5e9-848635b65061" containerName="revoke-keys"
Oct 31 01:54:48.609859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.609859    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d6020b-1edc-4c4b-b5e9-848635b65061" containerName="revoke-keys"
Oct 31 01:54:48.610069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.609890    5293 cpu_manager.go:414] "containerMap: removing container" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy"
Oct 31 01:54:48.610069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.609896    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy"
Oct 31 01:54:48.610069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.610024    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d6020b-1edc-4c4b-b5e9-848635b65061" containerName="revoke-keys"
Oct 31 01:54:48.610069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.610063    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c684b0-2883-49c6-b162-bab1e566f591" containerName="istio-proxy"
Oct 31 01:54:48.613624 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.613606    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.632445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.632388    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"]
Oct 31 01:54:48.661552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.661524    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-envoy\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.661683 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.661603    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-kube-api-access-n77gz\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-kube-api-access-n77gz\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.661683 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.661670    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-socket\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.661812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.661717    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c684b0-2883-49c6-b162-bab1e566f591-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c684b0-2883-49c6-b162-bab1e566f591-istiod-ca-cert\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.661812 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.661790    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-certs\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.662263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662053    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-credential-socket\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.662263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662156    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-certs" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:48.662263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662170    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-data\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.662263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662224    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c684b0-2883-49c6-b162-bab1e566f591-istiod-ca-cert" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:48.662522 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662284    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34c684b0-2883-49c6-b162-bab1e566f591-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c684b0-2883-49c6-b162-bab1e566f591-istio-podinfo\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.662522 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662310    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-credential-socket" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:48.662522 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662376    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-istio-token\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-istio-token\") pod \"34c684b0-2883-49c6-b162-bab1e566f591\" (UID: \"34c684b0-2883-49c6-b162-bab1e566f591\") "
Oct 31 01:54:48.662919 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662786    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-data" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:48.663011 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.662952    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-socket" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:48.663411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.663331    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c684b0-2883-49c6-b162-bab1e566f591-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.663411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.663354    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.663411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.663368    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.663411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.663381    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.664908 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.664886    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-envoy" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:48.665246 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.665228    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34c684b0-2883-49c6-b162-bab1e566f591-istio-podinfo" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:48.665406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.665386    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-istio-token" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:48.667456 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.667433    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-kube-api-access-n77gz" pod "34c684b0-2883-49c6-b162-bab1e566f591" (UID: "34c684b0-2883-49c6-b162-bab1e566f591"). InnerVolumeSpecName "kube-api-access-n77gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:48.833810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.821990    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.833810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.822085    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.833810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.822136    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7xw\" (UniqueName: \"kubernetes.io/projected/5ef4f950-cf22-4f06-a514-d051a1634334-kube-api-access-vp7xw\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.833810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.822590    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c684b0-2883-49c6-b162-bab1e566f591-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.833810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.822611    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.833810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.822625    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.833810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.822641    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n77gz\" (UniqueName: \"kubernetes.io/projected/34c684b0-2883-49c6-b162-bab1e566f591-kube-api-access-n77gz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.833810 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.822655    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c684b0-2883-49c6-b162-bab1e566f591-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:48.925287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.924307    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.925287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.924372    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.925287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.924423    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7xw\" (UniqueName: \"kubernetes.io/projected/5ef4f950-cf22-4f06-a514-d051a1634334-kube-api-access-vp7xw\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.926720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.926578    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.931009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.930960    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.948085 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.948053    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7xw\" (UniqueName: \"kubernetes.io/projected/5ef4f950-cf22-4f06-a514-d051a1634334-kube-api-access-vp7xw\") pod \"maas-api-revoke-sub-09606703-gwxls\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:48.953611 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:48.952259    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:49.242354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.242327    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"]
Oct 31 01:54:49.243382 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:54:49.243349    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef4f950_cf22_4f06_a514_d051a1634334.slice/crio-f722831b5ac8aa68765fc0ed76f2de758fe0c821e5eef2de8ab99ae3c8a33e61 WatchSource:0}: Error finding container f722831b5ac8aa68765fc0ed76f2de758fe0c821e5eef2de8ab99ae3c8a33e61: Status 404 returned error can't find the container with id f722831b5ac8aa68765fc0ed76f2de758fe0c821e5eef2de8ab99ae3c8a33e61
Oct 31 01:54:49.292791 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.292758    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls" event={"ID":"5ef4f950-cf22-4f06-a514-d051a1634334","Type":"ContainerStarted","Data":"f722831b5ac8aa68765fc0ed76f2de758fe0c821e5eef2de8ab99ae3c8a33e61"}
Oct 31 01:54:49.294223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.294199    5293 generic.go:356] "Generic (PLEG): container finished" podID="34c684b0-2883-49c6-b162-bab1e566f591" containerID="f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5" exitCode=0
Oct 31 01:54:49.294335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.294249    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"
Oct 31 01:54:49.294335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.294277    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" event={"ID":"34c684b0-2883-49c6-b162-bab1e566f591","Type":"ContainerDied","Data":"f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5"}
Oct 31 01:54:49.294335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.294328    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j" event={"ID":"34c684b0-2883-49c6-b162-bab1e566f591","Type":"ContainerDied","Data":"2c67b50fbbfd5fb494f72528fbefc3e53afb0430b12c67f8dc142702094744bc"}
Oct 31 01:54:49.294441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.294350    5293 scope.go:122] "RemoveContainer" containerID="f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5"
Oct 31 01:54:49.306983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.306957    5293 scope.go:122] "RemoveContainer" containerID="f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5"
Oct 31 01:54:49.307308 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:54:49.307287    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5\": container with ID starting with f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5 not found: ID does not exist" containerID="f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5"
Oct 31 01:54:49.307374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.307321    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5"} err="failed to get container status \"f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5\": rpc error: code = NotFound desc = could not find container \"f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5\": container with ID starting with f849e4033be36e76f16bc1cb152111144c46deb4edf69e2a1f42def46a16cef5 not found: ID does not exist"
Oct 31 01:54:49.320093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.320066    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"]
Oct 31 01:54:49.327687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.327664    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-dwb9j"]
Oct 31 01:54:49.568614 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.568588    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:49.638284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.638255    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-kube-api-access-2qks5\" (UniqueName: \"kubernetes.io/projected/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-kube-api-access-2qks5\") pod \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") "
Oct 31 01:54:49.638429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.638340    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-openshift-ca\") pod \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") "
Oct 31 01:54:49.638429 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.638396    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-xks-ca\") pod \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\" (UID: \"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453\") "
Oct 31 01:54:49.638908 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.638877    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-openshift-ca" pod "dd5bf6ea-fce6-47bb-a77f-37bd64ac7453" (UID: "dd5bf6ea-fce6-47bb-a77f-37bd64ac7453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:49.639295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.639271    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:49.641631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.641606    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-kube-api-access-2qks5" pod "dd5bf6ea-fce6-47bb-a77f-37bd64ac7453" (UID: "dd5bf6ea-fce6-47bb-a77f-37bd64ac7453"). InnerVolumeSpecName "kube-api-access-2qks5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:49.646290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.646238    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-xks-ca" pod "dd5bf6ea-fce6-47bb-a77f-37bd64ac7453" (UID: "dd5bf6ea-fce6-47bb-a77f-37bd64ac7453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:49.741106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.741076    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:49.741106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.741103    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qks5\" (UniqueName: \"kubernetes.io/projected/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453-kube-api-access-2qks5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:49.827244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:49.827156    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:54:50.175575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.175546    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c684b0-2883-49c6-b162-bab1e566f591" path="/var/lib/kubelet/pods/34c684b0-2883-49c6-b162-bab1e566f591/volumes"
Oct 31 01:54:50.314792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.314760    5293 generic.go:356] "Generic (PLEG): container finished" podID="5ef4f950-cf22-4f06-a514-d051a1634334" containerID="80a43f1cc906a6e9b92a07442ea4547f2ad2656b5ce0059c39c7c8443db16ecb" exitCode=0
Oct 31 01:54:50.315538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.315088    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls" event={"ID":"5ef4f950-cf22-4f06-a514-d051a1634334","Type":"ContainerDied","Data":"80a43f1cc906a6e9b92a07442ea4547f2ad2656b5ce0059c39c7c8443db16ecb"}
Oct 31 01:54:50.317955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.317934    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b" event={"ID":"dd5bf6ea-fce6-47bb-a77f-37bd64ac7453","Type":"ContainerDied","Data":"db6b16593492f02431560660c7c693fed84a495326b0cc1e9c9057749f15359b"}
Oct 31 01:54:50.318134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.318120    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db6b16593492f02431560660c7c693fed84a495326b0cc1e9c9057749f15359b"
Oct 31 01:54:50.318324 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.318304    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"
Oct 31 01:54:50.892546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.885811    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"]
Oct 31 01:54:50.892546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.888117    5293 cpu_manager.go:414] "containerMap: removing container" podUID="dd5bf6ea-fce6-47bb-a77f-37bd64ac7453" containerName="revoke-keys"
Oct 31 01:54:50.892546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.888140    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5bf6ea-fce6-47bb-a77f-37bd64ac7453" containerName="revoke-keys"
Oct 31 01:54:50.892546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.888372    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5bf6ea-fce6-47bb-a77f-37bd64ac7453" containerName="revoke-keys"
Oct 31 01:54:50.943253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.943225    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"]
Oct 31 01:54:50.943450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:50.943367    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.070741 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.070709    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxk7\" (UniqueName: \"kubernetes.io/projected/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-kube-api-access-xmxk7\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.070919 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.070867    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.070991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.070977    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.171818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.171786    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxk7\" (UniqueName: \"kubernetes.io/projected/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-kube-api-access-xmxk7\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.171990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.171899    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.171990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.171971    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.172589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.172561    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.174486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.174466    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.183238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.183213    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxk7\" (UniqueName: \"kubernetes.io/projected/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-kube-api-access-xmxk7\") pod \"maas-api-revoke-sub-16611557-zkw2q\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.277097 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.277070    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:51.515562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.515533    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:51.588406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.586103    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef4f950-cf22-4f06-a514-d051a1634334-kube-api-access-vp7xw\" (UniqueName: \"kubernetes.io/projected/5ef4f950-cf22-4f06-a514-d051a1634334-kube-api-access-vp7xw\") pod \"5ef4f950-cf22-4f06-a514-d051a1634334\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") "
Oct 31 01:54:51.588406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.586285    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-xks-ca\") pod \"5ef4f950-cf22-4f06-a514-d051a1634334\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") "
Oct 31 01:54:51.588406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.586441    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-openshift-ca\") pod \"5ef4f950-cf22-4f06-a514-d051a1634334\" (UID: \"5ef4f950-cf22-4f06-a514-d051a1634334\") "
Oct 31 01:54:51.588406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.588355    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-openshift-ca" pod "5ef4f950-cf22-4f06-a514-d051a1634334" (UID: "5ef4f950-cf22-4f06-a514-d051a1634334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:51.662586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.662553    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef4f950-cf22-4f06-a514-d051a1634334-kube-api-access-vp7xw" pod "5ef4f950-cf22-4f06-a514-d051a1634334" (UID: "5ef4f950-cf22-4f06-a514-d051a1634334"). InnerVolumeSpecName "kube-api-access-vp7xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:51.669072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.669016    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"]
Oct 31 01:54:51.676862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.676833    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-xks-ca" pod "5ef4f950-cf22-4f06-a514-d051a1634334" (UID: "5ef4f950-cf22-4f06-a514-d051a1634334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:51.690142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.690108    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.690276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.690157    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7xw\" (UniqueName: \"kubernetes.io/projected/5ef4f950-cf22-4f06-a514-d051a1634334-kube-api-access-vp7xw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.690276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:51.690172    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef4f950-cf22-4f06-a514-d051a1634334-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:52.449143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:52.444490    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls" event={"ID":"5ef4f950-cf22-4f06-a514-d051a1634334","Type":"ContainerDied","Data":"f722831b5ac8aa68765fc0ed76f2de758fe0c821e5eef2de8ab99ae3c8a33e61"}
Oct 31 01:54:52.449143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:52.444562    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f722831b5ac8aa68765fc0ed76f2de758fe0c821e5eef2de8ab99ae3c8a33e61"
Oct 31 01:54:52.449143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:52.444646    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"
Oct 31 01:54:52.449143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:52.447854    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q" event={"ID":"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e","Type":"ContainerStarted","Data":"f1362ec733f294253be05071db71f6642f88f225b745a294839eb384506e591d"}
Oct 31 01:54:52.449143 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:52.447905    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q" event={"ID":"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e","Type":"ContainerStarted","Data":"19dbab12789b810bd36aac022812d02819c03faa8080bc50970bf506c3ded613"}
Oct 31 01:54:52.487264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:52.487211    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q" podStartSLOduration=2.487197219 podStartE2EDuration="2.487197219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:52.484411559 +0000 UTC m=+2050.433059954" watchObservedRunningTime="2026-10-31 01:54:52.487197219 +0000 UTC m=+2050.435845610"
Oct 31 01:54:52.500936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:52.500906    5293 patch_prober.go:29] interesting pod/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:54:52.501140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:52.500965    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:54:53.483066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:53.479311    5293 generic.go:356] "Generic (PLEG): container finished" podID="e166a3a5-9fc7-448f-b0a7-f5792ae2d14e" containerID="f1362ec733f294253be05071db71f6642f88f225b745a294839eb384506e591d" exitCode=0
Oct 31 01:54:53.483066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:53.479575    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q" event={"ID":"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e","Type":"ContainerDied","Data":"f1362ec733f294253be05071db71f6642f88f225b745a294839eb384506e591d"}
Oct 31 01:54:54.622069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:54.621396    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:54.810521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:54.810432    5293 generic.go:356] "Generic (PLEG): container finished" podID="493b0f59-1286-4e4c-a47e-871418369fd9" containerID="b02874b823a70910ae0c34e85aed6534998a9f464c499357f9636ea315c25d78" exitCode=0
Oct 31 01:54:54.834558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:54.834526    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:54:54.858648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:54.858602    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" event={"ID":"493b0f59-1286-4e4c-a47e-871418369fd9","Type":"ContainerDied","Data":"b02874b823a70910ae0c34e85aed6534998a9f464c499357f9636ea315c25d78"}
Oct 31 01:54:54.859181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:54.859158    5293 scope.go:122] "RemoveContainer" containerID="b02874b823a70910ae0c34e85aed6534998a9f464c499357f9636ea315c25d78"
Oct 31 01:54:54.943067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:54.939589    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.015280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.011185    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-data\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.015280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.011367    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-istio-token\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-istio-token\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.015280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.011425    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-credential-socket\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.015280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.011476    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493b0f59-1286-4e4c-a47e-871418369fd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493b0f59-1286-4e4c-a47e-871418369fd9-istiod-ca-cert\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.015280 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.011654    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-kube-api-access-zhsnj\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-kube-api-access-zhsnj\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.039065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.027171    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-data" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:55.052912 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.052861    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493b0f59-1286-4e4c-a47e-871418369fd9-istiod-ca-cert" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:55.053804 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.053778    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-credential-socket" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:55.054871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.054850    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-envoy\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.055781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.055747    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/493b0f59-1286-4e4c-a47e-871418369fd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493b0f59-1286-4e4c-a47e-871418369fd9-istio-podinfo\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.061565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.061416    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-certs\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.084079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.061732    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-socket\") pod \"493b0f59-1286-4e4c-a47e-871418369fd9\" (UID: \"493b0f59-1286-4e4c-a47e-871418369fd9\") "
Oct 31 01:54:55.088817 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.088775    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-certs" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:55.089229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.089207    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-socket" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:55.136144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.133739    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.136144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.133772    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.136144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.133790    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.136144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.133806    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.136144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.133821    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493b0f59-1286-4e4c-a47e-871418369fd9-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.197066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.177465    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/493b0f59-1286-4e4c-a47e-871418369fd9-istio-podinfo" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:55.221067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.208987    5293 scope.go:122] "RemoveContainer" containerID="93c574bba911cde51456888ecfd597691ad14e0cb5e178fd68713ae896e86112"
Oct 31 01:54:55.221067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.208992    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-kube-api-access-zhsnj" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "kube-api-access-zhsnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:55.235073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.233184    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.245068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.243225    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-istio-token" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:55.245068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.243713    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 01:54:55.249207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.246568    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493b0f59-1286-4e4c-a47e-871418369fd9-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.249207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.246594    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.249207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.246610    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsnj\" (UniqueName: \"kubernetes.io/projected/493b0f59-1286-4e4c-a47e-871418369fd9-kube-api-access-zhsnj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.335125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.325221    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-envoy" pod "493b0f59-1286-4e4c-a47e-871418369fd9" (UID: "493b0f59-1286-4e4c-a47e-871418369fd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:55.433307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.368606    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493b0f59-1286-4e4c-a47e-871418369fd9-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.644068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.618831    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.672068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.622975    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.672068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.623018    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.672068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.645770    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:54:55.701769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.686423    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:55.701769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.686485    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.701769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.687373    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:54:55.756911 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.722271    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.756911 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.755143    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:54:55.914550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.914505    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:54:55.927416 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:55.927378    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.209187 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.197527    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.238102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.238032    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.240002 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.239971    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:56.240223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.240203    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.495169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.494321    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"
Oct 31 01:54:56.527220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.527188    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:54:56.530469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.530445    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:54:56.532306 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.532279    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p" event={"ID":"493b0f59-1286-4e4c-a47e-871418369fd9","Type":"ContainerDied","Data":"9164740a609817d6716a6b7ffc1b7995a26d88b75a169ce4ab0ecaf1ac73523a"}
Oct 31 01:54:56.532510 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.532493    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.620979    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"]
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.622054    5293 cpu_manager.go:414] "containerMap: removing container" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy"
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.622073    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy"
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.622086    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5ef4f950-cf22-4f06-a514-d051a1634334" containerName="revoke-keys"
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.622094    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef4f950-cf22-4f06-a514-d051a1634334" containerName="revoke-keys"
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.622124    5293 cpu_manager.go:414] "containerMap: removing container" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy"
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.622131    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy"
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.622300    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy"
Oct 31 01:54:56.624898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.622323    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef4f950-cf22-4f06-a514-d051a1634334" containerName="revoke-keys"
Oct 31 01:54:56.707340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.707307    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"]
Oct 31 01:54:56.707833 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.707415    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.712244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.712222    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:56.758436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.758359    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-xks-ca\") pod \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") "
Oct 31 01:54:56.758772 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.758750    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-kube-api-access-xmxk7\" (UniqueName: \"kubernetes.io/projected/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-kube-api-access-xmxk7\") pod \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") "
Oct 31 01:54:56.758975 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.758959    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-openshift-ca\") pod \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\" (UID: \"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e\") "
Oct 31 01:54:56.767063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.759641    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wtf\" (UniqueName: \"kubernetes.io/projected/c1791185-a8ab-4338-ba0f-a5d40aff5208-kube-api-access-q8wtf\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.767277 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.767256    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.769058 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.769013    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.774814 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.774790    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-openshift-ca" pod "e166a3a5-9fc7-448f-b0a7-f5792ae2d14e" (UID: "e166a3a5-9fc7-448f-b0a7-f5792ae2d14e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:56.781897 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.781834    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-xks-ca" pod "e166a3a5-9fc7-448f-b0a7-f5792ae2d14e" (UID: "e166a3a5-9fc7-448f-b0a7-f5792ae2d14e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:56.784082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.782154    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-kube-api-access-xmxk7" pod "e166a3a5-9fc7-448f-b0a7-f5792ae2d14e" (UID: "e166a3a5-9fc7-448f-b0a7-f5792ae2d14e"). InnerVolumeSpecName "kube-api-access-xmxk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:56.785729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.785701    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"]
Oct 31 01:54:56.876066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.872382    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.876066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.872518    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wtf\" (UniqueName: \"kubernetes.io/projected/c1791185-a8ab-4338-ba0f-a5d40aff5208-kube-api-access-q8wtf\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.876066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.872552    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.876066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.872689    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:56.876066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.872708    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:56.876066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.872724    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxk7\" (UniqueName: \"kubernetes.io/projected/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e-kube-api-access-xmxk7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:56.876066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.873493    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.887481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.876595    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:56.887481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.878061    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-77e2b5-openshift-default-64bb746b49-6hm8p"]
Oct 31 01:54:56.913656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:56.913055    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wtf\" (UniqueName: \"kubernetes.io/projected/c1791185-a8ab-4338-ba0f-a5d40aff5208-kube-api-access-q8wtf\") pod \"maas-api-revoke-sub-42475bd4-trm9g\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:57.074863 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:57.074778    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:57.369941 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:54:57.369906    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1791185_a8ab_4338_ba0f_a5d40aff5208.slice/crio-ae6b79c87576704e8a219c85cdaea4d8096d6f2a9c5bc99537345388132d9196 WatchSource:0}: Error finding container ae6b79c87576704e8a219c85cdaea4d8096d6f2a9c5bc99537345388132d9196: Status 404 returned error can't find the container with id ae6b79c87576704e8a219c85cdaea4d8096d6f2a9c5bc99537345388132d9196
Oct 31 01:54:57.370139 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:57.370116    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"]
Oct 31 01:54:57.506656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:57.506625    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"
Oct 31 01:54:57.506848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:57.506629    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q" event={"ID":"e166a3a5-9fc7-448f-b0a7-f5792ae2d14e","Type":"ContainerDied","Data":"19dbab12789b810bd36aac022812d02819c03faa8080bc50970bf506c3ded613"}
Oct 31 01:54:57.506848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:57.506766    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19dbab12789b810bd36aac022812d02819c03faa8080bc50970bf506c3ded613"
Oct 31 01:54:57.508716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:57.508689    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g" event={"ID":"c1791185-a8ab-4338-ba0f-a5d40aff5208","Type":"ContainerStarted","Data":"ae6b79c87576704e8a219c85cdaea4d8096d6f2a9c5bc99537345388132d9196"}
Oct 31 01:54:58.175320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:58.175286    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" path="/var/lib/kubelet/pods/493b0f59-1286-4e4c-a47e-871418369fd9/volumes"
Oct 31 01:54:58.522811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:58.522776    5293 generic.go:356] "Generic (PLEG): container finished" podID="c1791185-a8ab-4338-ba0f-a5d40aff5208" containerID="311a79b393e19338801f3a6aa6741e1e7db546333785ee82cd06fc8d6f401f4a" exitCode=0
Oct 31 01:54:58.522983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:58.522915    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g" event={"ID":"c1791185-a8ab-4338-ba0f-a5d40aff5208","Type":"ContainerDied","Data":"311a79b393e19338801f3a6aa6741e1e7db546333785ee82cd06fc8d6f401f4a"}
Oct 31 01:54:59.684214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.684187    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:54:59.724494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.723345    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-xks-ca\") pod \"c1791185-a8ab-4338-ba0f-a5d40aff5208\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") "
Oct 31 01:54:59.724494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.723494    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-openshift-ca\") pod \"c1791185-a8ab-4338-ba0f-a5d40aff5208\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") "
Oct 31 01:54:59.724494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.723553    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1791185-a8ab-4338-ba0f-a5d40aff5208-kube-api-access-q8wtf\" (UniqueName: \"kubernetes.io/projected/c1791185-a8ab-4338-ba0f-a5d40aff5208-kube-api-access-q8wtf\") pod \"c1791185-a8ab-4338-ba0f-a5d40aff5208\" (UID: \"c1791185-a8ab-4338-ba0f-a5d40aff5208\") "
Oct 31 01:54:59.725290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.725231    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-openshift-ca" pod "c1791185-a8ab-4338-ba0f-a5d40aff5208" (UID: "c1791185-a8ab-4338-ba0f-a5d40aff5208"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:59.727007 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.726973    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1791185-a8ab-4338-ba0f-a5d40aff5208-kube-api-access-q8wtf" pod "c1791185-a8ab-4338-ba0f-a5d40aff5208" (UID: "c1791185-a8ab-4338-ba0f-a5d40aff5208"). InnerVolumeSpecName "kube-api-access-q8wtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:59.728063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.728026    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-xks-ca" pod "c1791185-a8ab-4338-ba0f-a5d40aff5208" (UID: "c1791185-a8ab-4338-ba0f-a5d40aff5208"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:59.825080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.824981    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:59.825080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.825009    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1791185-a8ab-4338-ba0f-a5d40aff5208-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:59.825080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:54:59.825019    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8wtf\" (UniqueName: \"kubernetes.io/projected/c1791185-a8ab-4338-ba0f-a5d40aff5208-kube-api-access-q8wtf\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:00.418559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.418520    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"]
Oct 31 01:55:00.421067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.419412    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e166a3a5-9fc7-448f-b0a7-f5792ae2d14e" containerName="revoke-keys"
Oct 31 01:55:00.421067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.419435    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e166a3a5-9fc7-448f-b0a7-f5792ae2d14e" containerName="revoke-keys"
Oct 31 01:55:00.421067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.419453    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c1791185-a8ab-4338-ba0f-a5d40aff5208" containerName="revoke-keys"
Oct 31 01:55:00.421067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.419460    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1791185-a8ab-4338-ba0f-a5d40aff5208" containerName="revoke-keys"
Oct 31 01:55:00.421067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.419636    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e166a3a5-9fc7-448f-b0a7-f5792ae2d14e" containerName="revoke-keys"
Oct 31 01:55:00.421067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.419650    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1791185-a8ab-4338-ba0f-a5d40aff5208" containerName="revoke-keys"
Oct 31 01:55:00.421067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.419663    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="493b0f59-1286-4e4c-a47e-871418369fd9" containerName="istio-proxy"
Oct 31 01:55:00.424248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.424218    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.438332 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.434888    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-ed4dce-openshift-default-dockercfg-7n9zw\""
Oct 31 01:55:00.441711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.441545    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"]
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647305    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-socket\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647378    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-certs\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647411    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-credential-socket\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647443    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istiod-ca-cert\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647502    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-token\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647569    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-podinfo\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647610    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-data\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647654    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-envoy\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.663082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.647710    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbb2\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-kube-api-access-gmbb2\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.687823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.663915    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"
Oct 31 01:55:00.687823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.664107    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g" event={"ID":"c1791185-a8ab-4338-ba0f-a5d40aff5208","Type":"ContainerDied","Data":"ae6b79c87576704e8a219c85cdaea4d8096d6f2a9c5bc99537345388132d9196"}
Oct 31 01:55:00.687823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.664154    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae6b79c87576704e8a219c85cdaea4d8096d6f2a9c5bc99537345388132d9196"
Oct 31 01:55:00.748796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.748710    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-token\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.749111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.749085    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-podinfo\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.749407 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.749385    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-data\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.749553 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.749538    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-envoy\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.749682 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.749668    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbb2\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-kube-api-access-gmbb2\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.749970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.749950    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-socket\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.750132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.750118    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-certs\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.750249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.750236    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-credential-socket\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.750373 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.750356    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istiod-ca-cert\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.751350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.751327    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istiod-ca-cert\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.753726 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.752878    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-socket\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.753873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.753397    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-certs\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.759227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.759204    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-credential-socket\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.759679 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.759654    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-data\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.773485 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.773456    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-envoy\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.782711 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.782680    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-podinfo\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.789720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.789671    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-token\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:00.794314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:00.794290    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbb2\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-kube-api-access-gmbb2\") pod \"e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") " pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:01.045060 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:01.044959    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:01.338414 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:01.338381    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2088de2_2f63_4bc6_8e34_6fdaaa69593b.slice/crio-7c9599532e2c6e68ed6ef05cad797c2254245c4b31bc8a3a5c1d1cf7dc13d1ac WatchSource:0}: Error finding container 7c9599532e2c6e68ed6ef05cad797c2254245c4b31bc8a3a5c1d1cf7dc13d1ac: Status 404 returned error can't find the container with id 7c9599532e2c6e68ed6ef05cad797c2254245c4b31bc8a3a5c1d1cf7dc13d1ac
Oct 31 01:55:01.340432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:01.340399    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"]
Oct 31 01:55:01.341512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:01.341474    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:01.341642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:01.341552    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:01.341642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:01.341594    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:01.680186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:01.678919    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" event={"ID":"e2088de2-2f63-4bc6-8e34-6fdaaa69593b","Type":"ContainerStarted","Data":"1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0"}
Oct 31 01:55:01.680186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:01.678996    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" event={"ID":"e2088de2-2f63-4bc6-8e34-6fdaaa69593b","Type":"ContainerStarted","Data":"7c9599532e2c6e68ed6ef05cad797c2254245c4b31bc8a3a5c1d1cf7dc13d1ac"}
Oct 31 01:55:02.045360 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.045265    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:02.051207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.051183    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:02.082340 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.082292    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podStartSLOduration=2.082277107 podStartE2EDuration="2.082277107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:01.721538813 +0000 UTC m=+2059.670187206" watchObservedRunningTime="2026-10-31 01:55:02.082277107 +0000 UTC m=+2060.030925570"
Oct 31 01:55:02.688424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.688392    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:02.689278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.689255    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:55:02.812136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.812106    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"]
Oct 31 01:55:02.817096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.817075    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:02.820615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.820594    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:55:02.820746 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.820622    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:55:02.841272 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.840593    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"]
Oct 31 01:55:02.872600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.872566    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:55:02.872784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.872703    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:55:02.894023 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.893996    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:02.894183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.894105    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:02.894262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.894245    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntdt4\" (UniqueName: \"kubernetes.io/projected/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-kube-api-access-ntdt4\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:02.995994 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.995918    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:02.996168 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.996029    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdt4\" (UniqueName: \"kubernetes.io/projected/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-kube-api-access-ntdt4\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:02.996212 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.996199    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:02.996702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.996679    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:02.998734 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:02.998710    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:03.007471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:03.007448    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntdt4\" (UniqueName: \"kubernetes.io/projected/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-kube-api-access-ntdt4\") pod \"maas-api-revoke-sub-f934c6cf-fsdz7\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:03.150822 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:03.150790    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:03.439247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:03.439214    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"]
Oct 31 01:55:03.440607 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:03.440579    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaaabb2_1d46_4d9d_97ce_747cded4ae3a.slice/crio-54b42927821906e2dbd875c86661648f86c1733cec1ccefe461735e271300750 WatchSource:0}: Error finding container 54b42927821906e2dbd875c86661648f86c1733cec1ccefe461735e271300750: Status 404 returned error can't find the container with id 54b42927821906e2dbd875c86661648f86c1733cec1ccefe461735e271300750
Oct 31 01:55:03.697982 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:03.697898    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7" event={"ID":"baaaabb2-1d46-4d9d-97ce-747cded4ae3a","Type":"ContainerStarted","Data":"54b42927821906e2dbd875c86661648f86c1733cec1ccefe461735e271300750"}
Oct 31 01:55:04.717079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:04.717003    5293 generic.go:356] "Generic (PLEG): container finished" podID="baaaabb2-1d46-4d9d-97ce-747cded4ae3a" containerID="258862eb059a019f8a96af7e7b23c1015f14492ce5e6a52d3ae8a7bdcc0b2f80" exitCode=0
Oct 31 01:55:04.717737 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:04.717507    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7" event={"ID":"baaaabb2-1d46-4d9d-97ce-747cded4ae3a","Type":"ContainerDied","Data":"258862eb059a019f8a96af7e7b23c1015f14492ce5e6a52d3ae8a7bdcc0b2f80"}
Oct 31 01:55:06.140370 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.140343    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:06.313316 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.313218    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-openshift-ca\") pod \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") "
Oct 31 01:55:06.313502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.313397    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-kube-api-access-ntdt4\" (UniqueName: \"kubernetes.io/projected/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-kube-api-access-ntdt4\") pod \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") "
Oct 31 01:55:06.313502 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.313434    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-xks-ca\") pod \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\" (UID: \"baaaabb2-1d46-4d9d-97ce-747cded4ae3a\") "
Oct 31 01:55:06.313623 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.313593    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-openshift-ca" pod "baaaabb2-1d46-4d9d-97ce-747cded4ae3a" (UID: "baaaabb2-1d46-4d9d-97ce-747cded4ae3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:06.314086 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.314056    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:06.315706 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.315674    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-kube-api-access-ntdt4" pod "baaaabb2-1d46-4d9d-97ce-747cded4ae3a" (UID: "baaaabb2-1d46-4d9d-97ce-747cded4ae3a"). InnerVolumeSpecName "kube-api-access-ntdt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:06.315818 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.315715    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-xks-ca" pod "baaaabb2-1d46-4d9d-97ce-747cded4ae3a" (UID: "baaaabb2-1d46-4d9d-97ce-747cded4ae3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:55:06.416216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.416175    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntdt4\" (UniqueName: \"kubernetes.io/projected/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-kube-api-access-ntdt4\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:06.416216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.416206    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaaabb2-1d46-4d9d-97ce-747cded4ae3a-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:06.736947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.736914    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"
Oct 31 01:55:06.736947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.736933    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7" event={"ID":"baaaabb2-1d46-4d9d-97ce-747cded4ae3a","Type":"ContainerDied","Data":"54b42927821906e2dbd875c86661648f86c1733cec1ccefe461735e271300750"}
Oct 31 01:55:06.737257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:06.736981    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b42927821906e2dbd875c86661648f86c1733cec1ccefe461735e271300750"
Oct 31 01:55:07.905904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.905851    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"]
Oct 31 01:55:07.907113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.907085    5293 cpu_manager.go:414] "containerMap: removing container" podUID="baaaabb2-1d46-4d9d-97ce-747cded4ae3a" containerName="revoke-keys"
Oct 31 01:55:07.907113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.907114    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baaaabb2-1d46-4d9d-97ce-747cded4ae3a" containerName="revoke-keys"
Oct 31 01:55:07.907290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.907277    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="baaaabb2-1d46-4d9d-97ce-747cded4ae3a" containerName="revoke-keys"
Oct 31 01:55:07.911991 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.911970    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:07.918402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.917273    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:55:07.918402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.917338    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:55:07.918402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.917412    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:55:07.918402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.917453    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:55:07.920558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.920527    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"]
Oct 31 01:55:07.980510 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.980454    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:07.980730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.980529    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:07.980730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:07.980600    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjqh8\" (UniqueName: \"kubernetes.io/projected/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-kube-api-access-qjqh8\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:08.081444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.081413    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:08.081658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.081453    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:08.081658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.081490    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjqh8\" (UniqueName: \"kubernetes.io/projected/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-kube-api-access-qjqh8\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:08.082083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.082059    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:08.084142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.084118    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:08.094205 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.094180    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjqh8\" (UniqueName: \"kubernetes.io/projected/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-kube-api-access-qjqh8\") pod \"maas-api-revoke-sub-f8caac12-wjbfk\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:08.294974 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.294886    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:08.585889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.585844    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"]
Oct 31 01:55:08.586611 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:08.586585    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ee5480_48dd_4dbb_b146_e7c4ba29cca6.slice/crio-1908a046247fd3edc0126381743877e2e0a2b47c19f27a55d0a4ac6902fd2aae WatchSource:0}: Error finding container 1908a046247fd3edc0126381743877e2e0a2b47c19f27a55d0a4ac6902fd2aae: Status 404 returned error can't find the container with id 1908a046247fd3edc0126381743877e2e0a2b47c19f27a55d0a4ac6902fd2aae
Oct 31 01:55:08.757335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:08.757305    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk" event={"ID":"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6","Type":"ContainerStarted","Data":"1908a046247fd3edc0126381743877e2e0a2b47c19f27a55d0a4ac6902fd2aae"}
Oct 31 01:55:09.767940 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:09.767850    5293 generic.go:356] "Generic (PLEG): container finished" podID="c5ee5480-48dd-4dbb-b146-e7c4ba29cca6" containerID="fc5460bb836055da3e7c82ea0e85b1bfab7d6124bf843e8d20f805c83037d3b3" exitCode=0
Oct 31 01:55:09.768341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:09.767937    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk" event={"ID":"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6","Type":"ContainerDied","Data":"fc5460bb836055da3e7c82ea0e85b1bfab7d6124bf843e8d20f805c83037d3b3"}
Oct 31 01:55:11.046816 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.046735    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:11.061204 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.061173    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-kube-api-access-qjqh8\" (UniqueName: \"kubernetes.io/projected/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-kube-api-access-qjqh8\") pod \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") "
Oct 31 01:55:11.061354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.061238    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-xks-ca\") pod \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") "
Oct 31 01:55:11.061431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.061385    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-openshift-ca\") pod \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\" (UID: \"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6\") "
Oct 31 01:55:11.061733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.061705    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-openshift-ca" pod "c5ee5480-48dd-4dbb-b146-e7c4ba29cca6" (UID: "c5ee5480-48dd-4dbb-b146-e7c4ba29cca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:11.062209 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.061927    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:11.063601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.063576    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-xks-ca" pod "c5ee5480-48dd-4dbb-b146-e7c4ba29cca6" (UID: "c5ee5480-48dd-4dbb-b146-e7c4ba29cca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:55:11.063755 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.063735    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-kube-api-access-qjqh8" pod "c5ee5480-48dd-4dbb-b146-e7c4ba29cca6" (UID: "c5ee5480-48dd-4dbb-b146-e7c4ba29cca6"). InnerVolumeSpecName "kube-api-access-qjqh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:11.164121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.164088    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjqh8\" (UniqueName: \"kubernetes.io/projected/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-kube-api-access-qjqh8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:11.164121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.164121    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:11.781695 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:55:11.781660    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:55:11.786117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.786088    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"
Oct 31 01:55:11.786352 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.786084    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk" event={"ID":"c5ee5480-48dd-4dbb-b146-e7c4ba29cca6","Type":"ContainerDied","Data":"1908a046247fd3edc0126381743877e2e0a2b47c19f27a55d0a4ac6902fd2aae"}
Oct 31 01:55:11.786435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:11.786420    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1908a046247fd3edc0126381743877e2e0a2b47c19f27a55d0a4ac6902fd2aae"
Oct 31 01:55:13.073801 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.073765    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"]
Oct 31 01:55:13.074573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.074552    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c5ee5480-48dd-4dbb-b146-e7c4ba29cca6" containerName="revoke-keys"
Oct 31 01:55:13.074673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.074575    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ee5480-48dd-4dbb-b146-e7c4ba29cca6" containerName="revoke-keys"
Oct 31 01:55:13.074785 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.074771    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ee5480-48dd-4dbb-b146-e7c4ba29cca6" containerName="revoke-keys"
Oct 31 01:55:13.078625 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.078603    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.081645 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.081625    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:55:13.082083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.082058    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:55:13.082083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.082059    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:55:13.082252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.082116    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:55:13.091790 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.091762    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"]
Oct 31 01:55:13.111088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.102029    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99752b74-8d5c-4304-b904-49420758454b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.111088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.102210    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wx7\" (UniqueName: \"kubernetes.io/projected/99752b74-8d5c-4304-b904-49420758454b-kube-api-access-96wx7\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.111088 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.102264    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99752b74-8d5c-4304-b904-49420758454b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.204605 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.204569    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wx7\" (UniqueName: \"kubernetes.io/projected/99752b74-8d5c-4304-b904-49420758454b-kube-api-access-96wx7\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.204796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.204636    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99752b74-8d5c-4304-b904-49420758454b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.204984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.204968    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99752b74-8d5c-4304-b904-49420758454b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.205549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.205513    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99752b74-8d5c-4304-b904-49420758454b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.207802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.207779    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99752b74-8d5c-4304-b904-49420758454b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.216189 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.216168    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wx7\" (UniqueName: \"kubernetes.io/projected/99752b74-8d5c-4304-b904-49420758454b-kube-api-access-96wx7\") pod \"maas-api-revoke-sub-24b7f0b1-wtwjq\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.394177 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.394091    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:13.707738 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.707482    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"]
Oct 31 01:55:13.806501 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:13.806463    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq" event={"ID":"99752b74-8d5c-4304-b904-49420758454b","Type":"ContainerStarted","Data":"960ad7649fd754fa428c804092dcbb1573daf6d28c09bbaa62d462fe1583d446"}
Oct 31 01:55:14.844955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:14.844913    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq" event={"ID":"99752b74-8d5c-4304-b904-49420758454b","Type":"ContainerStarted","Data":"a0e6d241131a3ece5ab7d97628231a8aa1806e29fddec9a0584a3da80728d83a"}
Oct 31 01:55:14.864511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:14.864454    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq" podStartSLOduration=1.8644396460000001 podStartE2EDuration="1.864439646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:14.862456865 +0000 UTC m=+2072.811105257" watchObservedRunningTime="2026-10-31 01:55:14.864439646 +0000 UTC m=+2072.813088038"
Oct 31 01:55:15.055240 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:15.055174    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:55:15.893754 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:15.893717    5293 generic.go:356] "Generic (PLEG): container finished" podID="99752b74-8d5c-4304-b904-49420758454b" containerID="a0e6d241131a3ece5ab7d97628231a8aa1806e29fddec9a0584a3da80728d83a" exitCode=0
Oct 31 01:55:15.899829 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:15.894337    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq" event={"ID":"99752b74-8d5c-4304-b904-49420758454b","Type":"ContainerDied","Data":"a0e6d241131a3ece5ab7d97628231a8aa1806e29fddec9a0584a3da80728d83a"}
Oct 31 01:55:17.097140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.091192    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:55:17.522066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.509945    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.582336 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.582287    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.625003 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.624655    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.687066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.678599    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.737069 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.722806    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.750457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.750410    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.801119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.800971    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.898194 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.898144    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.982201 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:17.982154    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.092241 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:18.092134    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.092618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:18.092488    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.094167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:18.094006    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:55:18.214067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:18.180971    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.599062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:18.578321    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.627067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:18.604546    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.009077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.006406    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.120144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.064453    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.120144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.065246    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.120144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.067491    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 31 01:55:19.120144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.067809    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.120144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.080947    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.120144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.101355    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.120144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.118627    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:55:19.120144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.118700    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:19.231074 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.197602    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.244099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.238647    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.356185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.244627    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:19.514569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.514525    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cfzj2" podUID="3760c544-14ca-4dfd-9527-5054edd99b12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.676973 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.631235    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:19.809068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.735302    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.989296 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:19.989188    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:20.100095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.092983    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:20.100095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.095108    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:20.100095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.094191    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:55:20.100095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.094225    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:55:20.217110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.109274    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded"
Oct 31 01:55:20.217110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.117126    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:55:20.217110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.131539    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:20.217110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.202164    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:55:20.217110 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.211659    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:20.218224 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:55:20.218187    5293 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.017s"
Oct 31 01:55:20.314308 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.314207    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:55:20.326134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.317122    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:20.330183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.317161    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 01:55:20.401080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.398004    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99752b74-8d5c-4304-b904-49420758454b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99752b74-8d5c-4304-b904-49420758454b-maas-api-openshift-ca\") pod \"99752b74-8d5c-4304-b904-49420758454b\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") "
Oct 31 01:55:20.401080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.398076    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99752b74-8d5c-4304-b904-49420758454b-kube-api-access-96wx7\" (UniqueName: \"kubernetes.io/projected/99752b74-8d5c-4304-b904-49420758454b-kube-api-access-96wx7\") pod \"99752b74-8d5c-4304-b904-49420758454b\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") "
Oct 31 01:55:20.401080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.398211    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99752b74-8d5c-4304-b904-49420758454b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99752b74-8d5c-4304-b904-49420758454b-maas-api-xks-ca\") pod \"99752b74-8d5c-4304-b904-49420758454b\" (UID: \"99752b74-8d5c-4304-b904-49420758454b\") "
Oct 31 01:55:20.451091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.425723    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99752b74-8d5c-4304-b904-49420758454b-maas-api-openshift-ca" pod "99752b74-8d5c-4304-b904-49420758454b" (UID: "99752b74-8d5c-4304-b904-49420758454b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:20.631077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.593182    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99752b74-8d5c-4304-b904-49420758454b-maas-api-xks-ca" pod "99752b74-8d5c-4304-b904-49420758454b" (UID: "99752b74-8d5c-4304-b904-49420758454b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:55:20.631077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.612702    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99752b74-8d5c-4304-b904-49420758454b-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:20.654551 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.654501    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:20.754837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.754244    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99752b74-8d5c-4304-b904-49420758454b-kube-api-access-96wx7" pod "99752b74-8d5c-4304-b904-49420758454b" (UID: "99752b74-8d5c-4304-b904-49420758454b"). InnerVolumeSpecName "kube-api-access-96wx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:20.810098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.800501    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:55:20.810916 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.810892    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:55:20.811121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.811108    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:55:20.811249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.811238    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:55:20.811358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.811349    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:55:20.811474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.811462    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:55:20.811608 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.811594    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:55:20.811777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.811758    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:55:20.811910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.811899    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:55:20.812026 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.812017    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:55:20.812171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.812161    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:55:20.812287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.812278    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:55:20.812385 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.812377    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:55:20.835796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.782710    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96wx7\" (UniqueName: \"kubernetes.io/projected/99752b74-8d5c-4304-b904-49420758454b-kube-api-access-96wx7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:20.854063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.848995    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99752b74-8d5c-4304-b904-49420758454b-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:20.862840 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.862814    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:55:20.966067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.953912    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"
Oct 31 01:55:20.967001 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.966957    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq" event={"ID":"99752b74-8d5c-4304-b904-49420758454b","Type":"ContainerDied","Data":"960ad7649fd754fa428c804092dcbb1573daf6d28c09bbaa62d462fe1583d446"}
Oct 31 01:55:20.967251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:20.967231    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="960ad7649fd754fa428c804092dcbb1573daf6d28c09bbaa62d462fe1583d446"
Oct 31 01:55:21.145151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.138200    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:55:21.145151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.139258    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded"
Oct 31 01:55:21.156206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.156139    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:21.190118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.186259    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:21.220318 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.220274    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:21.346563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.346524    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:55:21.364175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.364122    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:55:21.398546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.383950    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:55:21.398546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.395550    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:55:21.398546 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.395584    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:55:21.425149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.425069    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:55:21.437302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.435649    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:55:21.437302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.436884    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 01:55:21.456195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:21.455115    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:55:23.916178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.916092    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"]
Oct 31 01:55:23.916928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.916904    5293 cpu_manager.go:414] "containerMap: removing container" podUID="99752b74-8d5c-4304-b904-49420758454b" containerName="revoke-keys"
Oct 31 01:55:23.917057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.916930    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99752b74-8d5c-4304-b904-49420758454b" containerName="revoke-keys"
Oct 31 01:55:23.917138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.917121    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="99752b74-8d5c-4304-b904-49420758454b" containerName="revoke-keys"
Oct 31 01:55:23.920990 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.920961    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:23.929704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.929683    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:55:23.934226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.932408    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:55:23.934226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.933690    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:55:23.934226 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.933880    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:55:23.939374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:23.939351    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"]
Oct 31 01:55:24.020827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.020795    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6wm\" (UniqueName: \"kubernetes.io/projected/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-kube-api-access-wr6wm\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.021013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.020871    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.021124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.021102    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.122550 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.122509    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6wm\" (UniqueName: \"kubernetes.io/projected/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-kube-api-access-wr6wm\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.122742 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.122598    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.122789 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.122754    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.123346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.123317    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.125275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.125250    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.134217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.134194    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6wm\" (UniqueName: \"kubernetes.io/projected/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-kube-api-access-wr6wm\") pod \"maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.238953 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.238866    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:24.533601 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:24.533572    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148ce1e3_b2bd_4c1b_8cbb_c600153e9516.slice/crio-49d28740387a990e2d9b02fd768e0c565b4655afe8a0aa5ef00a9200d645d38e WatchSource:0}: Error finding container 49d28740387a990e2d9b02fd768e0c565b4655afe8a0aa5ef00a9200d645d38e: Status 404 returned error can't find the container with id 49d28740387a990e2d9b02fd768e0c565b4655afe8a0aa5ef00a9200d645d38e
Oct 31 01:55:24.533715 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.533593    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"]
Oct 31 01:55:24.989997 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.989962    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw" event={"ID":"148ce1e3-b2bd-4c1b-8cbb-c600153e9516","Type":"ContainerStarted","Data":"fb324563c0f125f68bcf4a060af16bbf3fe0e88e20681d43d55d88564103819a"}
Oct 31 01:55:24.990493 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:24.990059    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw" event={"ID":"148ce1e3-b2bd-4c1b-8cbb-c600153e9516","Type":"ContainerStarted","Data":"49d28740387a990e2d9b02fd768e0c565b4655afe8a0aa5ef00a9200d645d38e"}
Oct 31 01:55:25.011628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.011579    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw" podStartSLOduration=2.01156566 podStartE2EDuration="2.01156566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:25.006830199 +0000 UTC m=+2082.955478591" watchObservedRunningTime="2026-10-31 01:55:25.01156566 +0000 UTC m=+2082.960214106"
Oct 31 01:55:25.240910 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.240879    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"]
Oct 31 01:55:25.245034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.245018    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:25.256700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.256670    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-46a67e\""
Oct 31 01:55:25.256836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.256713    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-46a67e-dockercfg-hsd2b\""
Oct 31 01:55:25.262152 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.262127    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"]
Oct 31 01:55:25.289723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.289692    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"]
Oct 31 01:55:25.295325 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.295306    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:25.314105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.313770    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"]
Oct 31 01:55:25.457068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.456739    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lqq\" (UniqueName: \"kubernetes.io/projected/0e78db94-ca55-4141-9a34-a5258f5fa0bc-kube-api-access-n5lqq\") pod \"payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g\" (UID: \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:25.457068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.456903    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vm7p\" (UniqueName: \"kubernetes.io/projected/0f201992-e685-4cbb-8ca9-60b12bd9d550-kube-api-access-8vm7p\") pod \"payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6\" (UID: \"0f201992-e685-4cbb-8ca9-60b12bd9d550\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:25.457068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.456942    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f201992-e685-4cbb-8ca9-60b12bd9d550-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6\" (UID: \"0f201992-e685-4cbb-8ca9-60b12bd9d550\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:25.457068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.456999    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78db94-ca55-4141-9a34-a5258f5fa0bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g\" (UID: \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:25.564298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.564208    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lqq\" (UniqueName: \"kubernetes.io/projected/0e78db94-ca55-4141-9a34-a5258f5fa0bc-kube-api-access-n5lqq\") pod \"payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g\" (UID: \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:25.565446 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.565417    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vm7p\" (UniqueName: \"kubernetes.io/projected/0f201992-e685-4cbb-8ca9-60b12bd9d550-kube-api-access-8vm7p\") pod \"payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6\" (UID: \"0f201992-e685-4cbb-8ca9-60b12bd9d550\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:25.566076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.566056    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f201992-e685-4cbb-8ca9-60b12bd9d550-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6\" (UID: \"0f201992-e685-4cbb-8ca9-60b12bd9d550\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:25.567355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.567335    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78db94-ca55-4141-9a34-a5258f5fa0bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g\" (UID: \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:25.585283 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.567254    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f201992-e685-4cbb-8ca9-60b12bd9d550-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6\" (UID: \"0f201992-e685-4cbb-8ca9-60b12bd9d550\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:25.587135 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.587102    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78db94-ca55-4141-9a34-a5258f5fa0bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g\" (UID: \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:25.591543 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.591457    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vm7p\" (UniqueName: \"kubernetes.io/projected/0f201992-e685-4cbb-8ca9-60b12bd9d550-kube-api-access-8vm7p\") pod \"payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6\" (UID: \"0f201992-e685-4cbb-8ca9-60b12bd9d550\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:25.595700 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.595652    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lqq\" (UniqueName: \"kubernetes.io/projected/0e78db94-ca55-4141-9a34-a5258f5fa0bc-kube-api-access-n5lqq\") pod \"payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g\" (UID: \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:25.627103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.625365    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:25.860870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:25.860787    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:26.059066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.052544    5293 generic.go:356] "Generic (PLEG): container finished" podID="148ce1e3-b2bd-4c1b-8cbb-c600153e9516" containerID="fb324563c0f125f68bcf4a060af16bbf3fe0e88e20681d43d55d88564103819a" exitCode=0
Oct 31 01:55:26.059066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.052779    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw" event={"ID":"148ce1e3-b2bd-4c1b-8cbb-c600153e9516","Type":"ContainerDied","Data":"fb324563c0f125f68bcf4a060af16bbf3fe0e88e20681d43d55d88564103819a"}
Oct 31 01:55:26.139703 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.139122    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"]
Oct 31 01:55:26.198406 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.198374    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"]
Oct 31 01:55:26.205365 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.205333    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.222350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.214587    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"]
Oct 31 01:55:26.222350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.222288    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-7a850e-openshift-default-dockercfg-mtlcf\""
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291262    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-podinfo\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291337    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-token\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291387    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-credential-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291420    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291463    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfpzm\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-kube-api-access-cfpzm\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291497    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a87c65c-8925-4bf1-952c-f380e42e2023-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291578    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-certs\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291650    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-data\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.292112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.291740    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-envoy\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.350325 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.350293    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"]
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.446971    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-envoy\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.447135    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-podinfo\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.447217    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-token\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.447270    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-credential-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.447306    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.447350    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfpzm\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-kube-api-access-cfpzm\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.447391    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a87c65c-8925-4bf1-952c-f380e42e2023-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.447481    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-certs\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.447559    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-data\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.448095    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-data\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.448783    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.450670    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-credential-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.451307    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a87c65c-8925-4bf1-952c-f380e42e2023-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.477068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.451870    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-certs\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.488089 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.479108    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-podinfo\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.497759 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.497697    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-token\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.554111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.545054    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfpzm\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-kube-api-access-cfpzm\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.554111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.545780    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-envoy\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.662846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.662812    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:26.977236 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:26.977208    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a87c65c_8925_4bf1_952c_f380e42e2023.slice/crio-98f4a7bd3b41d9b2a74862dfa22eb3b8bf38c9cc27f462e74b6b37fcc3910fd8 WatchSource:0}: Error finding container 98f4a7bd3b41d9b2a74862dfa22eb3b8bf38c9cc27f462e74b6b37fcc3910fd8: Status 404 returned error can't find the container with id 98f4a7bd3b41d9b2a74862dfa22eb3b8bf38c9cc27f462e74b6b37fcc3910fd8
Oct 31 01:55:26.977471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.977448    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"]
Oct 31 01:55:26.980077 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.980008    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:26.980178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.980111    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:26.980178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:26.980141    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:27.065346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.065310    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6" event={"ID":"0f201992-e685-4cbb-8ca9-60b12bd9d550","Type":"ContainerStarted","Data":"4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0"}
Oct 31 01:55:27.065784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.065383    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6" event={"ID":"0f201992-e685-4cbb-8ca9-60b12bd9d550","Type":"ContainerStarted","Data":"443f36381d10f33968258e27b75ac6e0bba58b4887cf7c0b9869b681be3de9db"}
Oct 31 01:55:27.065784 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.065415    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:27.067843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.067646    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" event={"ID":"8a87c65c-8925-4bf1-952c-f380e42e2023","Type":"ContainerStarted","Data":"d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e"}
Oct 31 01:55:27.067843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.067695    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" event={"ID":"8a87c65c-8925-4bf1-952c-f380e42e2023","Type":"ContainerStarted","Data":"98f4a7bd3b41d9b2a74862dfa22eb3b8bf38c9cc27f462e74b6b37fcc3910fd8"}
Oct 31 01:55:27.069708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.069687    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g" event={"ID":"0e78db94-ca55-4141-9a34-a5258f5fa0bc","Type":"ContainerStarted","Data":"67f969368eb723c23239757e95e1275ea6b73f2770434a4e71cb53b02daff669"}
Oct 31 01:55:27.069830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.069723    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g" event={"ID":"0e78db94-ca55-4141-9a34-a5258f5fa0bc","Type":"ContainerStarted","Data":"9fda3c4343d996941842f25a53fab3a503114577ab77e910be11a30588435c30"}
Oct 31 01:55:27.069830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.069769    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:27.088538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.088478    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6" podStartSLOduration=2.088460387 podStartE2EDuration="2.088460387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:27.082168477 +0000 UTC m=+2085.030816873" watchObservedRunningTime="2026-10-31 01:55:27.088460387 +0000 UTC m=+2085.037108782"
Oct 31 01:55:27.113096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.106709    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g" podStartSLOduration=2.1066946 podStartE2EDuration="2.1066946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:27.103449216 +0000 UTC m=+2085.052097610" watchObservedRunningTime="2026-10-31 01:55:27.1066946 +0000 UTC m=+2085.055342993"
Oct 31 01:55:27.133140 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.133081    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podStartSLOduration=1.1330635550000001 podStartE2EDuration="1.133063555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:27.126236824 +0000 UTC m=+2085.074885218" watchObservedRunningTime="2026-10-31 01:55:27.133063555 +0000 UTC m=+2085.081711947"
Oct 31 01:55:27.386694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.386673    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:27.486852 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.486815    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-kube-api-access-wr6wm\" (UniqueName: \"kubernetes.io/projected/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-kube-api-access-wr6wm\") pod \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") "
Oct 31 01:55:27.487109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.486882    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-xks-ca\") pod \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") "
Oct 31 01:55:27.487109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.487003    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-openshift-ca\") pod \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\" (UID: \"148ce1e3-b2bd-4c1b-8cbb-c600153e9516\") "
Oct 31 01:55:27.488431 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.488402    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-openshift-ca" pod "148ce1e3-b2bd-4c1b-8cbb-c600153e9516" (UID: "148ce1e3-b2bd-4c1b-8cbb-c600153e9516"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:27.490330 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.490305    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-xks-ca" pod "148ce1e3-b2bd-4c1b-8cbb-c600153e9516" (UID: "148ce1e3-b2bd-4c1b-8cbb-c600153e9516"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:55:27.490409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.490345    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-kube-api-access-wr6wm" pod "148ce1e3-b2bd-4c1b-8cbb-c600153e9516" (UID: "148ce1e3-b2bd-4c1b-8cbb-c600153e9516"). InnerVolumeSpecName "kube-api-access-wr6wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:27.589223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.589143    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr6wm\" (UniqueName: \"kubernetes.io/projected/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-kube-api-access-wr6wm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.589223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.589174    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.589223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.589188    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148ce1e3-b2bd-4c1b-8cbb-c600153e9516-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.663937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:27.663901    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:28.079473 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:28.079443    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"
Oct 31 01:55:28.079871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:28.079493    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw" event={"ID":"148ce1e3-b2bd-4c1b-8cbb-c600153e9516","Type":"ContainerDied","Data":"49d28740387a990e2d9b02fd768e0c565b4655afe8a0aa5ef00a9200d645d38e"}
Oct 31 01:55:28.079871 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:28.079537    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d28740387a990e2d9b02fd768e0c565b4655afe8a0aa5ef00a9200d645d38e"
Oct 31 01:55:28.669243 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:28.669211    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:29.088610 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:29.088533    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:29.089575 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:29.089550    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:55:34.180564 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:34.180341    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"]
Oct 31 01:55:35.517112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.517026    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"]
Oct 31 01:55:35.517904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.517883    5293 cpu_manager.go:414] "containerMap: removing container" podUID="148ce1e3-b2bd-4c1b-8cbb-c600153e9516" containerName="revoke-keys"
Oct 31 01:55:35.518010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.517905    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148ce1e3-b2bd-4c1b-8cbb-c600153e9516" containerName="revoke-keys"
Oct 31 01:55:35.518105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.518095    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="148ce1e3-b2bd-4c1b-8cbb-c600153e9516" containerName="revoke-keys"
Oct 31 01:55:35.521846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.521826    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:35.528347 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.528233    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-2e6bfd-dockercfg-6xgzg\""
Oct 31 01:55:35.534463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.534442    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"]
Oct 31 01:55:35.587265 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.587237    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2e6bfd\""
Oct 31 01:55:35.589295 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.589267    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945eac35-ca1c-4d53-974a-bc2d7df38685-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l\" (UID: \"945eac35-ca1c-4d53-974a-bc2d7df38685\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:35.589608 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.589589    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvm5q\" (UniqueName: \"kubernetes.io/projected/945eac35-ca1c-4d53-974a-bc2d7df38685-kube-api-access-vvm5q\") pod \"payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l\" (UID: \"945eac35-ca1c-4d53-974a-bc2d7df38685\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:35.694531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.694489    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvm5q\" (UniqueName: \"kubernetes.io/projected/945eac35-ca1c-4d53-974a-bc2d7df38685-kube-api-access-vvm5q\") pod \"payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l\" (UID: \"945eac35-ca1c-4d53-974a-bc2d7df38685\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:35.694883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.694861    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945eac35-ca1c-4d53-974a-bc2d7df38685-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l\" (UID: \"945eac35-ca1c-4d53-974a-bc2d7df38685\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:35.717066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.711527    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945eac35-ca1c-4d53-974a-bc2d7df38685-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l\" (UID: \"945eac35-ca1c-4d53-974a-bc2d7df38685\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:35.785115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.784505    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvm5q\" (UniqueName: \"kubernetes.io/projected/945eac35-ca1c-4d53-974a-bc2d7df38685-kube-api-access-vvm5q\") pod \"payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l\" (UID: \"945eac35-ca1c-4d53-974a-bc2d7df38685\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:35.849061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:35.848319    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:36.264787 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.264757    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"]
Oct 31 01:55:36.265776 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:36.265753    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945eac35_ca1c_4d53_974a_bc2d7df38685.slice/crio-1db8d5c55152b82e1d6edc26d474ef21fd401a822ac8e63cdb5b38a85cae3149 WatchSource:0}: Error finding container 1db8d5c55152b82e1d6edc26d474ef21fd401a822ac8e63cdb5b38a85cae3149: Status 404 returned error can't find the container with id 1db8d5c55152b82e1d6edc26d474ef21fd401a822ac8e63cdb5b38a85cae3149
Oct 31 01:55:36.525065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.524838    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"]
Oct 31 01:55:36.546111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.530508    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.550284 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.547901    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"]
Oct 31 01:55:36.665439 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.665396    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.665633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.665544    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.665633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.665588    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dade4288-429f-4934-801f-19a34ad4eb14-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.665633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.665608    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.665633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.665632    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.665890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.665782    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.665979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.665960    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.666057 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.666008    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dade4288-429f-4934-801f-19a34ad4eb14-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.666128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.666115    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6cc\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-kube-api-access-zt6cc\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.767649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.767612    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.767880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.767726    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.767880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.767765    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dade4288-429f-4934-801f-19a34ad4eb14-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.767880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.767795    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.767880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.767828    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.767880 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.767875    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.768016    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.768075    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dade4288-429f-4934-801f-19a34ad4eb14-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.768105    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6cc\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-kube-api-access-zt6cc\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.768220    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.768273    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.768490    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.768526    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.769640    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dade4288-429f-4934-801f-19a34ad4eb14-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.771353    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.784756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.771817    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dade4288-429f-4934-801f-19a34ad4eb14-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.814066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.805176    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.831065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.820848    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6cc\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-kube-api-access-zt6cc\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:36.871066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:36.870431    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:37.303535 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.302895    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"]
Oct 31 01:55:37.314252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.314223    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.413496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.413466    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"]
Oct 31 01:55:37.414409 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414377    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-data\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.414532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414428    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6p7\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-kube-api-access-fd6p7\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.414532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414476    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-istio-token\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.414532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414509    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201f0b07-540b-453c-a8c2-990aca029322-istio-podinfo\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.414710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414563    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-credential-socket\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.414710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414606    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201f0b07-540b-453c-a8c2-990aca029322-istiod-ca-cert\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.414710 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414701    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-certs\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.414864 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414729    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-socket\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.414864 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.414753    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-envoy\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.439902 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.439871    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-6d7f5d-openshift-default-dockercfg-txj5d\""
Oct 31 01:55:37.452571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.451744    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l" event={"ID":"945eac35-ca1c-4d53-974a-bc2d7df38685","Type":"ContainerStarted","Data":"614f0b02914173bc4cce4cb4fc37a5fb143c09acd1a06fc77c285b58a94fd92c"}
Oct 31 01:55:37.452571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.451795    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l" event={"ID":"945eac35-ca1c-4d53-974a-bc2d7df38685","Type":"ContainerStarted","Data":"1db8d5c55152b82e1d6edc26d474ef21fd401a822ac8e63cdb5b38a85cae3149"}
Oct 31 01:55:37.452571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.451816    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:37.513329 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.513271    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l" podStartSLOduration=2.513252006 podStartE2EDuration="2.513252006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:37.505377008 +0000 UTC m=+2095.454025401" watchObservedRunningTime="2026-10-31 01:55:37.513252006 +0000 UTC m=+2095.461900401"
Oct 31 01:55:37.516018 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.515985    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-data\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.516668 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.516650    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6p7\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-kube-api-access-fd6p7\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.517183 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.517164    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-istio-token\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.517561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.517542    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201f0b07-540b-453c-a8c2-990aca029322-istio-podinfo\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.518167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.518147    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-credential-socket\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.518631 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.518616    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201f0b07-540b-453c-a8c2-990aca029322-istiod-ca-cert\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.519992 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.519971    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-certs\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.520423 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.520404    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-socket\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.520545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.520529    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-envoy\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.521383 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.516593    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-data\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.522025 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.520354    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-certs\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.522190 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.519623    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201f0b07-540b-453c-a8c2-990aca029322-istiod-ca-cert\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.522294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.521775    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-socket\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.523379 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.518537    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-credential-socket\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.544890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.544124    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-envoy\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.544890 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.544673    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201f0b07-540b-453c-a8c2-990aca029322-istio-podinfo\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.553104 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.550341    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6p7\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-kube-api-access-fd6p7\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.632989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.631967    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-istio-token\") pod \"e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") " pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.806381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.806035    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:37.895080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.894611    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"]
Oct 31 01:55:37.935931 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.934352    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:37.935931 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.934464    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:37.935931 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:37.934524    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:38.114327 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:38.114250    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 01:55:38.132549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:38.132521    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 01:55:38.706858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:38.706822    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"]
Oct 31 01:55:38.713065 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:38.708401    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201f0b07_540b_453c_a8c2_990aca029322.slice/crio-3f38ffde6d3a992401dcb616e72927be22c92a8c9557f92727a7cc9b107fa4ed WatchSource:0}: Error finding container 3f38ffde6d3a992401dcb616e72927be22c92a8c9557f92727a7cc9b107fa4ed: Status 404 returned error can't find the container with id 3f38ffde6d3a992401dcb616e72927be22c92a8c9557f92727a7cc9b107fa4ed
Oct 31 01:55:38.846714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:38.846288    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:38.858813 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:38.858675    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:38.859783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:38.859577    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:55:38.966008 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:38.965964    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" event={"ID":"dade4288-429f-4934-801f-19a34ad4eb14","Type":"ContainerStarted","Data":"42cce3deca1c43146d5f158663c746b0a77be25530a885cd119a18996b3d797a"}
Oct 31 01:55:39.196307 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:39.182102    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm" podUID="137b698a-1305-45df-ad72-b1cf414d31ec" containerName="payload-pre-processing" containerID="cri-o://a89bc306aa776c79cf37b3ef942cc7fdffe58921b9cea4c91cc12fb4d7b781a7" gracePeriod=25
Oct 31 01:55:39.265772 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:39.228072    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:39.481482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:39.457003    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:39.503065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:39.499186    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:39.648532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:39.648449    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:39.727276 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:39.726261    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:39.829608 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:39.829578    5293 logging.go:55] [core] [Channel #2696 SubChannel #2697]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.210:50052", ServerName: "10.133.0.210:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.210:50052: connect: connection refused"
Oct 31 01:55:39.944180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:39.944133    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:40.042027 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.041931    5293 generic.go:356] "Generic (PLEG): container finished" podID="137b698a-1305-45df-ad72-b1cf414d31ec" containerID="a89bc306aa776c79cf37b3ef942cc7fdffe58921b9cea4c91cc12fb4d7b781a7" exitCode=0
Oct 31 01:55:40.042353 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.042326    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm" event={"ID":"137b698a-1305-45df-ad72-b1cf414d31ec","Type":"ContainerDied","Data":"a89bc306aa776c79cf37b3ef942cc7fdffe58921b9cea4c91cc12fb4d7b781a7"}
Oct 31 01:55:40.054615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.053265    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" event={"ID":"201f0b07-540b-453c-a8c2-990aca029322","Type":"ContainerStarted","Data":"4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3"}
Oct 31 01:55:40.054615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.053332    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" event={"ID":"201f0b07-540b-453c-a8c2-990aca029322","Type":"ContainerStarted","Data":"3f38ffde6d3a992401dcb616e72927be22c92a8c9557f92727a7cc9b107fa4ed"}
Oct 31 01:55:40.071109 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.057661    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" event={"ID":"dade4288-429f-4934-801f-19a34ad4eb14","Type":"ContainerStarted","Data":"473a0dbcfbecf5ab9f425302bea2efadc6c4478cbca7080ac0d71e341f09c041"}
Oct 31 01:55:40.310162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.237135    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:40.310162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.284539    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podStartSLOduration=3.2845127 podStartE2EDuration="3.2845127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:40.101394114 +0000 UTC m=+2098.050042508" watchObservedRunningTime="2026-10-31 01:55:40.2845127 +0000 UTC m=+2098.233161090"
Oct 31 01:55:40.509194 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.509145    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:40.612380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.612283    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:55:40.700141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.672375    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:55:40.704091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.703263    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:40.736094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.729895    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:55:40.736094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.733528    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podStartSLOduration=4.7335066040000005 podStartE2EDuration="4.733506604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:40.240360485 +0000 UTC m=+2098.189008913" watchObservedRunningTime="2026-10-31 01:55:40.733506604 +0000 UTC m=+2098.682154996"
Oct 31 01:55:40.775078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.774438    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:55:40.775078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.774852    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:40.826326 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.826288    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:55:40.826601 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.826574    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:55:40.858065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.840949    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm" podUID="137b698a-1305-45df-ad72-b1cf414d31ec" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.210:50052\" within 1s: context deadline exceeded"
Oct 31 01:55:40.874255 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.874121    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:40.932680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:40.932634    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:41.031385 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.024772    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:55:41.068843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.067947    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:41.090179 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.090133    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:41.183062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.176443    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:41.263064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.261344    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 01:55:41.263064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.261399    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 01:55:41.284483 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:55:41.284448    5293 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.057s"
Oct 31 01:55:41.284823 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.284802    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:41.285023 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.285008    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:55:41.285282 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.285266    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:55:41.285404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.285391    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:41.285563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.285550    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:55:41.302072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.285696    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:55:41.316584 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.316561    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:55:41.529125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.527749    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:55:41.615216 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.615173    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:41.615859 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.615838    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:55:41.623010 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.622974    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm" event={"ID":"137b698a-1305-45df-ad72-b1cf414d31ec","Type":"ContainerDied","Data":"19fbdaecda9f7a495285f1bb4ee8166d44fb85cbb40b36814a4b4f239b50ab77"}
Oct 31 01:55:41.643987 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.643951    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19fbdaecda9f7a495285f1bb4ee8166d44fb85cbb40b36814a4b4f239b50ab77"
Oct 31 01:55:41.684178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.684129    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:41.685750 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.685724    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:55:41.750134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.745019    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:41.777673 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.745990    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:41.927155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.927030    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 01:55:41.927155 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.927112    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 01:55:42.020157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:41.993626    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:55:42.028251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.028224    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:55:42.062066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.033291    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:55:42.122517 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.122481    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:42.203154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.202398    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:42.260073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.259292    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:42.304068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.276858    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137b698a-1305-45df-ad72-b1cf414d31ec-kube-api-access-tpt4v\" (UniqueName: \"kubernetes.io/projected/137b698a-1305-45df-ad72-b1cf414d31ec-kube-api-access-tpt4v\") pod \"137b698a-1305-45df-ad72-b1cf414d31ec\" (UID: \"137b698a-1305-45df-ad72-b1cf414d31ec\") "
Oct 31 01:55:42.316181 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.316141    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137b698a-1305-45df-ad72-b1cf414d31ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b698a-1305-45df-ad72-b1cf414d31ec-plugins-config-volume\") pod \"137b698a-1305-45df-ad72-b1cf414d31ec\" (UID: \"137b698a-1305-45df-ad72-b1cf414d31ec\") "
Oct 31 01:55:42.417475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.407710    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 31 01:55:42.417475 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.417251    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137b698a-1305-45df-ad72-b1cf414d31ec-kube-api-access-tpt4v" pod "137b698a-1305-45df-ad72-b1cf414d31ec" (UID: "137b698a-1305-45df-ad72-b1cf414d31ec"). InnerVolumeSpecName "kube-api-access-tpt4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:42.500647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.500553    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpt4v\" (UniqueName: \"kubernetes.io/projected/137b698a-1305-45df-ad72-b1cf414d31ec-kube-api-access-tpt4v\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:42.503722 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.503331    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137b698a-1305-45df-ad72-b1cf414d31ec-plugins-config-volume" pod "137b698a-1305-45df-ad72-b1cf414d31ec" (UID: "137b698a-1305-45df-ad72-b1cf414d31ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:42.557669 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.557602    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:55:42.676819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.676729    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:42.681342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.681307    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:42.706063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.693582    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137b698a-1305-45df-ad72-b1cf414d31ec-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:42.706063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.698403    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:42.750085 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:55:42.738405    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:55:42.752351 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.752268    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:42.770070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.767072    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:55:42.770070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.767138    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:55:42.770070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.767215    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:55:42.770070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.767361    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 01:55:42.770070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.767398    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:55:42.770070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.767443    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 01:55:42.770070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.767472    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:55:42.775056 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.775011    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:42.775419 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.775390    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"
Oct 31 01:55:42.775566 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.775550    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:55:42.778896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.778863    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:55:42.871388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.871352    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 01:55:42.871579 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:42.871406    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 01:55:43.054900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.054821    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"]
Oct 31 01:55:43.060628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.060600    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbff33-5db9c986f9-bjfqm"]
Oct 31 01:55:43.871783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.871740    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 01:55:43.872309 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.871815    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 01:55:43.891539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.891508    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"]
Oct 31 01:55:43.892251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.892234    5293 cpu_manager.go:414] "containerMap: removing container" podUID="137b698a-1305-45df-ad72-b1cf414d31ec" containerName="payload-pre-processing"
Oct 31 01:55:43.892251 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.892251    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137b698a-1305-45df-ad72-b1cf414d31ec" containerName="payload-pre-processing"
Oct 31 01:55:43.892371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.892354    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="137b698a-1305-45df-ad72-b1cf414d31ec" containerName="payload-pre-processing"
Oct 31 01:55:43.907369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.907341    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:43.959073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.956628    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"]
Oct 31 01:55:43.959073 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.958919    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6d7f5d\""
Oct 31 01:55:43.959341 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:43.959092    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6d7f5d-dockercfg-7hfz5\""
Oct 31 01:55:44.135120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.076290    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4d72e-1061-4b38-bc36-ec9727d25e00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz\" (UID: \"58d4d72e-1061-4b38-bc36-ec9727d25e00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:44.135120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.076366    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gk5z\" (UniqueName: \"kubernetes.io/projected/58d4d72e-1061-4b38-bc36-ec9727d25e00-kube-api-access-8gk5z\") pod \"payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz\" (UID: \"58d4d72e-1061-4b38-bc36-ec9727d25e00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:44.183300 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.183263    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4d72e-1061-4b38-bc36-ec9727d25e00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz\" (UID: \"58d4d72e-1061-4b38-bc36-ec9727d25e00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:44.183618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.183597    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gk5z\" (UniqueName: \"kubernetes.io/projected/58d4d72e-1061-4b38-bc36-ec9727d25e00-kube-api-access-8gk5z\") pod \"payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz\" (UID: \"58d4d72e-1061-4b38-bc36-ec9727d25e00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:44.185725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.185682    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4d72e-1061-4b38-bc36-ec9727d25e00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz\" (UID: \"58d4d72e-1061-4b38-bc36-ec9727d25e00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:44.232314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.232227    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gk5z\" (UniqueName: \"kubernetes.io/projected/58d4d72e-1061-4b38-bc36-ec9727d25e00-kube-api-access-8gk5z\") pod \"payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz\" (UID: \"58d4d72e-1061-4b38-bc36-ec9727d25e00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:44.286063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.249429    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:44.495545 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.495506    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137b698a-1305-45df-ad72-b1cf414d31ec" path="/var/lib/kubelet/pods/137b698a-1305-45df-ad72-b1cf414d31ec/volumes"
Oct 31 01:55:44.871797 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.871746    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 01:55:44.872200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:44.871794    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 01:55:45.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:45.211814    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"]
Oct 31 01:55:45.226211 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:55:45.221511    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d4d72e_1061_4b38_bc36_ec9727d25e00.slice/crio-372f2f59cd494c463bf436651b0668dd9aad9246b1b78929fd90e2ce02f05ac6 WatchSource:0}: Error finding container 372f2f59cd494c463bf436651b0668dd9aad9246b1b78929fd90e2ce02f05ac6: Status 404 returned error can't find the container with id 372f2f59cd494c463bf436651b0668dd9aad9246b1b78929fd90e2ce02f05ac6
Oct 31 01:55:45.464729 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:45.464517    5293 scope.go:122] "RemoveContainer" containerID="c142401263d60b18bca2f3e37fa8aefb6e4fc5cfee08fc5d10dcfb291ad429cb"
Oct 31 01:55:45.519559 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:45.516291    5293 scope.go:122] "RemoveContainer" containerID="7a6487c3b79e99b91f6e5625be6f8fe347122af2e99ddf2ac2eea9f387db2832"
Oct 31 01:55:45.897117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:45.893344    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 01:55:45.897117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:45.893403    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 01:55:46.061487 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:46.061317    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz" event={"ID":"58d4d72e-1061-4b38-bc36-ec9727d25e00","Type":"ContainerStarted","Data":"5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596"}
Oct 31 01:55:46.061487 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:46.061402    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz" event={"ID":"58d4d72e-1061-4b38-bc36-ec9727d25e00","Type":"ContainerStarted","Data":"372f2f59cd494c463bf436651b0668dd9aad9246b1b78929fd90e2ce02f05ac6"}
Oct 31 01:55:46.063745 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:46.062692    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:55:46.110033 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:46.109967    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz" podStartSLOduration=3.109946327 podStartE2EDuration="3.109946327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:46.109060856 +0000 UTC m=+2104.057709249" watchObservedRunningTime="2026-10-31 01:55:46.109946327 +0000 UTC m=+2104.058594722"
Oct 31 01:55:46.176835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:46.161148    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"]
Oct 31 01:55:46.225159 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:46.222078    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-dbff33-664fw"]
Oct 31 01:55:46.871474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:46.871006    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 01:55:46.871474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:46.871082    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 01:55:47.871485 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:47.871443    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 01:55:47.871904 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:47.871514    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 01:55:48.176536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:48.176498    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148ce1e3-b2bd-4c1b-8cbb-c600153e9516" path="/var/lib/kubelet/pods/148ce1e3-b2bd-4c1b-8cbb-c600153e9516/volumes"
Oct 31 01:55:48.972822 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:48.972795    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 01:55:49.507199 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:49.507172    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:49.507725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:49.507709    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:49.508609 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:49.508581    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 01:55:56.709267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:56.709229    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:55:58.082761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:55:58.082734    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 01:56:02.660253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.660217    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"]
Oct 31 01:56:02.668369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.668022    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.685937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.683379    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c8b702-openshift-default-dockercfg-m92wm\""
Oct 31 01:56:02.692992 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.692966    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"]
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.824492    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-certs\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.824592    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-data\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.824647    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-token\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.824724    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb25a85b-918d-4810-b90c-8b1d2e21a369-istiod-ca-cert\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.824791    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhbh\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-kube-api-access-tkhbh\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.824860    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-credential-socket\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.825007    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-envoy\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.825079    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-socket\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.827062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.825186    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-podinfo\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.944507    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-data\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.944582    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-token\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.944650    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb25a85b-918d-4810-b90c-8b1d2e21a369-istiod-ca-cert\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.944707    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhbh\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-kube-api-access-tkhbh\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.944771    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-credential-socket\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.944892    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-envoy\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.944973    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-socket\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.945087    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-podinfo\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:02.959133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:02.945142    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-certs\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.013092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.002705    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-socket\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.014970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.014938    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-credential-socket\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.066169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.016526    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb25a85b-918d-4810-b90c-8b1d2e21a369-istiod-ca-cert\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.079102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.017313    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-certs\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.097174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.052596    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-data\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.097389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.087181    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-token\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.097485 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.090244    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhbh\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-kube-api-access-tkhbh\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.110061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.105649    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-envoy\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.125654 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.125521    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-podinfo\") pod \"e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") " pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.284716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.284623    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:03.953313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.953275    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"]
Oct 31 01:56:03.977553 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:56:03.977518    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb25a85b_918d_4810_b90c_8b1d2e21a369.slice/crio-6d392cf2698516644ef2155b4139da93abbe1df702dc005a34b5b7fbac989549 WatchSource:0}: Error finding container 6d392cf2698516644ef2155b4139da93abbe1df702dc005a34b5b7fbac989549: Status 404 returned error can't find the container with id 6d392cf2698516644ef2155b4139da93abbe1df702dc005a34b5b7fbac989549
Oct 31 01:56:03.982218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.982176    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:03.982540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.982515    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:03.982704 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:03.982683    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:04.832696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.832665    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"]
Oct 31 01:56:04.838530 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.837841    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.843410 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.843384    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-46a67e-openshift-default-dockercfg-ttzhz\""
Oct 31 01:56:04.857659 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.857545    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"]
Oct 31 01:56:04.969989 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.969956    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.970361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.970016    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65xv6\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-kube-api-access-65xv6\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.970361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.970177    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.970361 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.970216    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.970486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.970373    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.970486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.970451    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.970486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.970475    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.970587 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.970552    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.970587 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.970578    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:04.988805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.988771    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" event={"ID":"eb25a85b-918d-4810-b90c-8b1d2e21a369","Type":"ContainerStarted","Data":"02637c1b02660536d09ff65f6307af3959ee8a1dd3c9faf5ee358b0a909480b2"}
Oct 31 01:56:04.988979 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:04.988835    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" event={"ID":"eb25a85b-918d-4810-b90c-8b1d2e21a369","Type":"ContainerStarted","Data":"6d392cf2698516644ef2155b4139da93abbe1df702dc005a34b5b7fbac989549"}
Oct 31 01:56:05.012549 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.012487    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" podStartSLOduration=3.01246764 podStartE2EDuration="3.01246764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:05.010423544 +0000 UTC m=+2122.959071936" watchObservedRunningTime="2026-10-31 01:56:05.01246764 +0000 UTC m=+2122.961116033"
Oct 31 01:56:05.072141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072052    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072259    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072355 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072328    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65xv6\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-kube-api-access-65xv6\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072395 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072368    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072407    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072422    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072515    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072572    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072612 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072601    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072666    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072725 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072685    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.072942 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072908    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.073078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.072952    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.073178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.073157    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.074955 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.074933    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.075275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.075246    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.083848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.083826    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.083981 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.083848    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65xv6\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-kube-api-access-65xv6\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.164202 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.164170    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:05.300063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.291559    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:05.316920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.316474    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:05.766718 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:56:05.766681    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e32b58_f094_4f6c_beec_7ff70cf7efaa.slice/crio-91974652c04569d91f919710287c3879cb834c2b08b263a36bb7a93ad279579d WatchSource:0}: Error finding container 91974652c04569d91f919710287c3879cb834c2b08b263a36bb7a93ad279579d: Status 404 returned error can't find the container with id 91974652c04569d91f919710287c3879cb834c2b08b263a36bb7a93ad279579d
Oct 31 01:56:05.767836 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.767812    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"]
Oct 31 01:56:05.769532 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.769493    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:05.769630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.769580    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:05.769630 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:05.769623    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:06.002617 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:06.002573    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" event={"ID":"20e32b58-f094-4f6c-beec-7ff70cf7efaa","Type":"ContainerStarted","Data":"d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d"}
Oct 31 01:56:06.003319 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:06.002646    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" event={"ID":"20e32b58-f094-4f6c-beec-7ff70cf7efaa","Type":"ContainerStarted","Data":"91974652c04569d91f919710287c3879cb834c2b08b263a36bb7a93ad279579d"}
Oct 31 01:56:06.003715 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:06.003697    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:06.004838 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:06.004813    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:56:06.033217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:06.031899    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podStartSLOduration=2.031880145 podStartE2EDuration="2.031880145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:06.02425702 +0000 UTC m=+2123.972905416" watchObservedRunningTime="2026-10-31 01:56:06.031880145 +0000 UTC m=+2123.980528538"
Oct 31 01:56:06.164764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:06.164735    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:06.165936 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:06.165897    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 31 01:56:06.166087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:06.165978    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 31 01:56:07.169460 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:07.169416    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 31 01:56:07.170083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:07.170027    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 31 01:56:08.165253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:08.165210    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 31 01:56:08.165457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:08.165296    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 31 01:56:09.220246 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:09.190063    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 31 01:56:09.220246 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:09.190122    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 31 01:56:10.164724 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:10.164688    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 31 01:56:10.165028 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:10.164755    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 31 01:56:11.166070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:11.165407    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 31 01:56:11.166070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:11.166034    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 31 01:56:11.778493 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:56:11.778390    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:56:12.169285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:12.169236    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 31 01:56:12.169804 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:12.169320    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 31 01:56:12.266530 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:12.266493    5293 generic.go:356] "Generic (PLEG): container finished" podID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerID="d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e" exitCode=0
Oct 31 01:56:12.266708 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:12.266541    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" event={"ID":"8a87c65c-8925-4bf1-952c-f380e42e2023","Type":"ContainerDied","Data":"d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e"}
Oct 31 01:56:12.267024 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:12.267010    5293 scope.go:122] "RemoveContainer" containerID="d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e"
Oct 31 01:56:12.268767 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:12.268734    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:12.268875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:12.268803    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:12.268875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:12.268834    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:13.170114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.170086    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:13.170589 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.170570    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:13.171376 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.171354    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 01:56:13.283094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.283014    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" event={"ID":"8a87c65c-8925-4bf1-952c-f380e42e2023","Type":"ContainerStarted","Data":"1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2"}
Oct 31 01:56:13.486178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.479885    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"]
Oct 31 01:56:13.493287 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.493261    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.525118 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.525089    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"]
Oct 31 01:56:13.537108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.534252    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"]
Oct 31 01:56:13.545522 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.544889    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.550632 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.549506    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"]
Oct 31 01:56:13.589330 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.589305    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.589488 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.589344    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.589619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.589592    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99da1e81-f392-47a3-962a-cec0233e0288-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.589735 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.589716    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99da1e81-f392-47a3-962a-cec0233e0288-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.589875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.589847    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.589917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.589895    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgdf\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-kube-api-access-mhgdf\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.589963 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.589950    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.589997 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.589976    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.590059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.590008    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.663665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.663630    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:56:13.665244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.665217    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 01:56:13.665426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.665267    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 01:56:13.691932 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.691897    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.692102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.691944    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.692102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692061    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99da1e81-f392-47a3-962a-cec0233e0288-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692192    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.692249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692225    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.692339 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692255    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.692378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692334    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692368    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgdf\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-kube-api-access-mhgdf\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692394    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9z4\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-kube-api-access-2h9z4\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.692486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692416    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692471    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91a20441-02fd-4989-b638-2bde95779d23-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.692653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692512    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692653 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692577    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692746    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692822 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692794    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692844    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99da1e81-f392-47a3-962a-cec0233e0288-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.692883 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692868    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.692998 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.692906    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.693099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.693032    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.693180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.693101    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.693180 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.693075    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99da1e81-f392-47a3-962a-cec0233e0288-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.693274 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.693259    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.693313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.693262    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91a20441-02fd-4989-b638-2bde95779d23-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.695564 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.695544    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.695914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.695894    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99da1e81-f392-47a3-962a-cec0233e0288-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.726924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.726868    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.728096 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.727863    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgdf\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-kube-api-access-mhgdf\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.794723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.794589    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.794893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.794841    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91a20441-02fd-4989-b638-2bde95779d23-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.794964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.794891    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.794964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.794917    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795031    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795080    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795110    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795176    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9z4\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-kube-api-access-2h9z4\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795209    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91a20441-02fd-4989-b638-2bde95779d23-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795256    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795441 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795393    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795558    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795651    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.795678 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.795665    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91a20441-02fd-4989-b638-2bde95779d23-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.812486 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.812380    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.829811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.829771    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91a20441-02fd-4989-b638-2bde95779d23-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.832030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.832002    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:13.833920 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.833896    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9z4\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-kube-api-access-2h9z4\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.836558 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.836539    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:13.869367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:13.869338    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:14.090298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.089542    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:56:14.090834 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.090720    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"]
Oct 31 01:56:14.124186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.124157    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:14.138756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.138689    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"]
Oct 31 01:56:14.213375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.213342    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e57add\""
Oct 31 01:56:14.235125 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.224318    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e57add-dockercfg-fdnfw\""
Oct 31 01:56:14.235539 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.226178    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5wb\" (UniqueName: \"kubernetes.io/projected/fe286755-5d6e-43ec-b73f-dc46b359b9ab-kube-api-access-wj5wb\") pod \"payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq\" (UID: \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:14.243674 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.235956    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe286755-5d6e-43ec-b73f-dc46b359b9ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq\" (UID: \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:14.430677 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.430636    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe286755-5d6e-43ec-b73f-dc46b359b9ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq\" (UID: \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:14.435075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.430874    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5wb\" (UniqueName: \"kubernetes.io/projected/fe286755-5d6e-43ec-b73f-dc46b359b9ab-kube-api-access-wj5wb\") pod \"payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq\" (UID: \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:14.435075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.432240    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe286755-5d6e-43ec-b73f-dc46b359b9ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq\" (UID: \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:14.525091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.522625    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:14.525091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.522752    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:14.525091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.522798    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:14.666870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.664715    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5wb\" (UniqueName: \"kubernetes.io/projected/fe286755-5d6e-43ec-b73f-dc46b359b9ab-kube-api-access-wj5wb\") pod \"payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq\" (UID: \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:14.666870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.664826    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 01:56:14.666870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.664872    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 01:56:14.666870 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.665989    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"]
Oct 31 01:56:14.671451 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.671407    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"]
Oct 31 01:56:14.675934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.675885    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:14.676151 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.676114    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:14.676366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.676241    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:14.758520 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:14.758490    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:15.228253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.228165    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"]
Oct 31 01:56:15.667648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.667606    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 01:56:15.667866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.667671    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 01:56:15.731185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.721718    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" event={"ID":"91a20441-02fd-4989-b638-2bde95779d23","Type":"ContainerStarted","Data":"0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc"}
Oct 31 01:56:15.731185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.721816    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" event={"ID":"91a20441-02fd-4989-b638-2bde95779d23","Type":"ContainerStarted","Data":"c151df719a6a675f0dcab9de644d522b24dd6309182bb401a813f7f8d5e7d319"}
Oct 31 01:56:15.731185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.730601    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq" event={"ID":"fe286755-5d6e-43ec-b73f-dc46b359b9ab","Type":"ContainerStarted","Data":"ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584"}
Oct 31 01:56:15.731185 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.730661    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq" event={"ID":"fe286755-5d6e-43ec-b73f-dc46b359b9ab","Type":"ContainerStarted","Data":"c1b25b75291bc1b524b205c37060c27585bc6fc991aeec6a6c82958cea0af8fe"}
Oct 31 01:56:15.733696 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.731615    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:15.756078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.737761    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" event={"ID":"99da1e81-f392-47a3-962a-cec0233e0288","Type":"ContainerStarted","Data":"421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35"}
Oct 31 01:56:15.756078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.737810    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" event={"ID":"99da1e81-f392-47a3-962a-cec0233e0288","Type":"ContainerStarted","Data":"be22453cda01f70ba4c8217bfe16893069b27390ccb191ac9ba5c3827f554f47"}
Oct 31 01:56:15.771067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.770801    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podStartSLOduration=2.770782126 podStartE2EDuration="2.770782126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:15.769679169 +0000 UTC m=+2133.718327585" watchObservedRunningTime="2026-10-31 01:56:15.770782126 +0000 UTC m=+2133.719430519"
Oct 31 01:56:15.840080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.839757    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:15.843465 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.843387    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podStartSLOduration=2.84336711 podStartE2EDuration="2.84336711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:15.815747089 +0000 UTC m=+2133.764395481" watchObservedRunningTime="2026-10-31 01:56:15.84336711 +0000 UTC m=+2133.792015504"
Oct 31 01:56:15.852665 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.852614    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq" podStartSLOduration=1.852596929 podStartE2EDuration="1.852596929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:15.839703995 +0000 UTC m=+2133.788352388" watchObservedRunningTime="2026-10-31 01:56:15.852596929 +0000 UTC m=+2133.801245322"
Oct 31 01:56:15.879066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.875609    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:15.879066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.875722    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:15.879066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:15.875761    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:16.867092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:16.848819    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 01:56:16.867819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:16.867780    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 01:56:17.061066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.055931    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:17.061066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.055994    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:17.072072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.062753    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:17.072072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.062810    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:17.594059 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.593805    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:17.680619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.674439    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 01:56:17.680619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.674922    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 01:56:17.848117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.836423    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:17.848117 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.836488    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:17.868866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.868160    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:17.894075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.879206    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:17.894075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.879263    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:17.894075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.888141    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:17.894075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.889490    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:56:17.995163 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:17.995122    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:18.054167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:18.054121    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:18.674067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:18.670158    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 01:56:18.674067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:18.670223    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 01:56:18.819068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:18.786407    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:18.946615 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:18.918414    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:19.032156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:18.919294    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:19.058167 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.058123    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:19.058401 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.058128    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.058805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.032379    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:19.080087 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.059629    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:56:19.266435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.265163    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:56:19.266435 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.265526    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.466192    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.480070 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.466682    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:56:19.535124 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.534282    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.667062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.663425    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 01:56:19.667062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.663488    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 01:56:19.718065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.712155    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.832537 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.832447    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:19.832903 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.832864    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:19.877581 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.877540    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:19.877878 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.877833    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:19.886828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.886790    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.894707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.894661    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 31 01:56:19.895200 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.894601    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.948184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.948137    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.949368 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:19.949341    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:56:20.075578 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.075392    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:56:20.267191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.267140    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 01:56:20.267600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.267565    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:20.268331 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.268303    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 01:56:20.640114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.393995    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:56:20.640114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.394080    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:56:20.640114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.603191    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 01:56:20.640114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.608224    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:56:20.659446 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.659412    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 01:56:20.704325 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.675729    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 01:56:20.704593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.704556    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 01:56:20.712076 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.700324    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:20.843496 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.843460    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:20.843766 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.843740    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:20.844266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.844243    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 01:56:20.871947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.871915    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:20.872218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.872193    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:20.888331 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.888293    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:20.888773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.888755    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:56:20.896107 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.895660    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:20.925358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.907871    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:56:20.956463 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.955173    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:20.982224 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:20.981216    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 01:56:21.069106 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.068388    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 01:56:21.090464 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.089589    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 01:56:21.721583 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.720643    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:21.721583 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.721290    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:56:21.758067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.748686    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:21.831067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.826948    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 01:56:21.831067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.829013    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 01:56:21.873236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.873189    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:21.873555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.873507    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:21.875732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.874847    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:21.875732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.874888    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:21.875732 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.875360    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 01:56:21.889595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.889327    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:56:21.992186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:21.992085    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 01:56:22.839030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:22.838993    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:22.839596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:22.839561    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:22.870055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:22.870000    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:22.870252 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:22.870086    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:23.835064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:23.834360    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:23.835064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:23.834421    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:23.870756 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:23.870714    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:23.870974 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:23.870779    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:24.846518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:24.834963    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:56:24.846518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:24.835027    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:56:24.906372 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:24.906336    5293 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 01:56:24.906675 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:24.906635    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 01:56:25.840951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:25.840919    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:25.841507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:25.841490    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:25.842382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:25.842355    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 01:56:25.874462 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:25.874426    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:25.874837 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:25.874700    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:25.875388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:25.875372    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 01:56:26.910655 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:26.910609    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:56:28.441067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:28.436348    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 01:56:32.054162 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:32.054125    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"]
Oct 31 01:56:32.077115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:32.077073    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373189b1-sjdgn"]
Oct 31 01:56:32.182438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:32.182407    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034d73b8-d39e-4a17-9370-b46a9a058934" path="/var/lib/kubelet/pods/034d73b8-d39e-4a17-9370-b46a9a058934/volumes"
Oct 31 01:56:41.551805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.551773    5293 generic.go:356] "Generic (PLEG): container finished" podID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerID="1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0" exitCode=0
Oct 31 01:56:41.552440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.551848    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" event={"ID":"e2088de2-2f63-4bc6-8e34-6fdaaa69593b","Type":"ContainerDied","Data":"1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0"}
Oct 31 01:56:41.552440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.552402    5293 scope.go:122] "RemoveContainer" containerID="1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0"
Oct 31 01:56:41.553888 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.553857    5293 generic.go:356] "Generic (PLEG): container finished" podID="201f0b07-540b-453c-a8c2-990aca029322" containerID="4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3" exitCode=0
Oct 31 01:56:41.554017 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.553925    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" event={"ID":"201f0b07-540b-453c-a8c2-990aca029322","Type":"ContainerDied","Data":"4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3"}
Oct 31 01:56:41.554291 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.554263    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:41.554346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.554333    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:41.554385 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.554368    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:41.554420 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.554391    5293 scope.go:122] "RemoveContainer" containerID="4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3"
Oct 31 01:56:41.556105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.556078    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:41.556207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.556149    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:41.556207 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:41.556187    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:56:41.909064 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:56:41.908681    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:56:42.576512 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:42.576471    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" event={"ID":"e2088de2-2f63-4bc6-8e34-6fdaaa69593b","Type":"ContainerStarted","Data":"41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7"}
Oct 31 01:56:42.585772 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:42.585738    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" event={"ID":"201f0b07-540b-453c-a8c2-990aca029322","Type":"ContainerStarted","Data":"bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a"}
Oct 31 01:56:42.788066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:42.785656    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:56:42.811068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:42.809552    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:56:42.811068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:42.809693    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:56:42.811068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:42.809747    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:56:43.050927 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:43.049805    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:56:43.072065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:43.071724    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:56:43.072065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:43.071785    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:56:43.820063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:43.813498    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:56:43.820063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:43.813577    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:56:44.047066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:44.046569    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:56:44.047066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:44.046663    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:56:44.807853 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:44.807406    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:56:44.807853 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:44.807470    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:56:45.054191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:45.052188    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:56:45.054191 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:45.052292    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:56:45.658067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:45.646703    5293 scope.go:122] "RemoveContainer" containerID="b8622d5e86f700778121b9e88f2b52f6204d93b231060ad7d37a088ed3958370"
Oct 31 01:56:45.673514 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:45.673475    5293 scope.go:122] "RemoveContainer" containerID="1298ef85051a60bffa36edb61e157bba1477a974622ee0cd4762375c7911a024"
Oct 31 01:56:45.815511 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:45.815468    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:56:45.815805 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:45.815774    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:56:46.185362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:46.185328    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:56:46.185962 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:46.185927    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:56:46.808499 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:46.808455    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:56:46.808807 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:46.808780    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:56:47.052095 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.052058    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:56:47.052381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.052349    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:56:47.535769 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.535207    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 01:56:47.691266 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.690153    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:56:47.709647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.709606    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:47.822144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.821508    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:56:47.822144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.821568    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:56:47.831227 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.831149    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:47.969498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:47.969453    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:48.049563 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:48.049523    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:56:48.049846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:48.049815    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:56:48.810469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:48.810431    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:56:48.811083 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:48.811031    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:56:49.162720 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:49.162689    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:56:49.162984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:49.162956    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:56:49.807845 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:49.807806    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 31 01:56:49.808175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:49.808140    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 31 01:56:49.867288 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:49.867241    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:50.047426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:50.047378    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:56:50.047620 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:50.047452    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:56:50.833078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:50.832338    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:56:50.851728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:50.851695    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 01:56:51.050993 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:51.050962    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:56:51.052536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:51.052516    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 01:56:56.903105 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:56.899888    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"]
Oct 31 01:56:56.908993 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:56.908963    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:56.920482 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:56.920449    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"]
Oct 31 01:56:56.924178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:56.920876    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:56:56.924490 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:56.920939    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:56:56.986472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:56.986437    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:56:56.986660 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:56.986537    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:56:57.022244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.022212    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8kb\" (UniqueName: \"kubernetes.io/projected/a82e88a4-5e8e-473a-ab77-48050d2274a3-kube-api-access-ks8kb\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.022573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.022555    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.022846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.022826    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.125263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.125225    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8kb\" (UniqueName: \"kubernetes.io/projected/a82e88a4-5e8e-473a-ab77-48050d2274a3-kube-api-access-ks8kb\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.125263 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.125267    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.125528 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.125350    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.126144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.126121    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.127964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.127937    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.136658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.136634    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8kb\" (UniqueName: \"kubernetes.io/projected/a82e88a4-5e8e-473a-ab77-48050d2274a3-kube-api-access-ks8kb\") pod \"maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.264843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.264748    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:56:57.579338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:57.579298    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"]
Oct 31 01:56:57.581445 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:56:57.581416    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82e88a4_5e8e_473a_ab77_48050d2274a3.slice/crio-b106ca860cb08bb095a9bdbd130c2cb559ff191c1fa5c5651a816681784b16dd WatchSource:0}: Error finding container b106ca860cb08bb095a9bdbd130c2cb559ff191c1fa5c5651a816681784b16dd: Status 404 returned error can't find the container with id b106ca860cb08bb095a9bdbd130c2cb559ff191c1fa5c5651a816681784b16dd
Oct 31 01:56:58.032005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:58.031966    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z" event={"ID":"a82e88a4-5e8e-473a-ab77-48050d2274a3","Type":"ContainerStarted","Data":"874f6cabacb88195dfa54a1920fc866e9458e2f428b5cc152d3ac0a505c2c1dd"}
Oct 31 01:56:58.032521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:58.032030    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z" event={"ID":"a82e88a4-5e8e-473a-ab77-48050d2274a3","Type":"ContainerStarted","Data":"b106ca860cb08bb095a9bdbd130c2cb559ff191c1fa5c5651a816681784b16dd"}
Oct 31 01:56:58.058761 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:58.058711    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z" podStartSLOduration=2.058696506 podStartE2EDuration="2.058696506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:58.054275855 +0000 UTC m=+2176.002924245" watchObservedRunningTime="2026-10-31 01:56:58.058696506 +0000 UTC m=+2176.007344944"
Oct 31 01:56:59.043378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:59.043337    5293 generic.go:356] "Generic (PLEG): container finished" podID="a82e88a4-5e8e-473a-ab77-48050d2274a3" containerID="874f6cabacb88195dfa54a1920fc866e9458e2f428b5cc152d3ac0a505c2c1dd" exitCode=0
Oct 31 01:56:59.043862 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:56:59.043400    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z" event={"ID":"a82e88a4-5e8e-473a-ab77-48050d2274a3","Type":"ContainerDied","Data":"874f6cabacb88195dfa54a1920fc866e9458e2f428b5cc152d3ac0a505c2c1dd"}
Oct 31 01:57:00.054144 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.054114    5293 generic.go:356] "Generic (PLEG): container finished" podID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerID="02637c1b02660536d09ff65f6307af3959ee8a1dd3c9faf5ee358b0a909480b2" exitCode=0
Oct 31 01:57:00.054622 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.054184    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" event={"ID":"eb25a85b-918d-4810-b90c-8b1d2e21a369","Type":"ContainerDied","Data":"02637c1b02660536d09ff65f6307af3959ee8a1dd3c9faf5ee358b0a909480b2"}
Oct 31 01:57:00.055225 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.054719    5293 scope.go:122] "RemoveContainer" containerID="02637c1b02660536d09ff65f6307af3959ee8a1dd3c9faf5ee358b0a909480b2"
Oct 31 01:57:00.056695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.056664    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:57:00.056773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.056738    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:57:00.056820 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.056771    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:57:00.445192 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.445164    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:57:00.461778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.461750    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82e88a4-5e8e-473a-ab77-48050d2274a3-kube-api-access-ks8kb\" (UniqueName: \"kubernetes.io/projected/a82e88a4-5e8e-473a-ab77-48050d2274a3-kube-api-access-ks8kb\") pod \"a82e88a4-5e8e-473a-ab77-48050d2274a3\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") "
Oct 31 01:57:00.461966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.461854    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-xks-ca\") pod \"a82e88a4-5e8e-473a-ab77-48050d2274a3\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") "
Oct 31 01:57:00.461966 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.461920    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-openshift-ca\") pod \"a82e88a4-5e8e-473a-ab77-48050d2274a3\" (UID: \"a82e88a4-5e8e-473a-ab77-48050d2274a3\") "
Oct 31 01:57:00.462896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.462866    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-openshift-ca" pod "a82e88a4-5e8e-473a-ab77-48050d2274a3" (UID: "a82e88a4-5e8e-473a-ab77-48050d2274a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:00.463597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.463184    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:00.465538 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.465514    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82e88a4-5e8e-473a-ab77-48050d2274a3-kube-api-access-ks8kb" pod "a82e88a4-5e8e-473a-ab77-48050d2274a3" (UID: "a82e88a4-5e8e-473a-ab77-48050d2274a3"). InnerVolumeSpecName "kube-api-access-ks8kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:00.467764 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.467740    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-xks-ca" pod "a82e88a4-5e8e-473a-ab77-48050d2274a3" (UID: "a82e88a4-5e8e-473a-ab77-48050d2274a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:00.564951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.564916    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8kb\" (UniqueName: \"kubernetes.io/projected/a82e88a4-5e8e-473a-ab77-48050d2274a3-kube-api-access-ks8kb\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:00.564951 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:00.564950    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82e88a4-5e8e-473a-ab77-48050d2274a3-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:01.064471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:01.064441    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"
Oct 31 01:57:01.064471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:01.064463    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z" event={"ID":"a82e88a4-5e8e-473a-ab77-48050d2274a3","Type":"ContainerDied","Data":"b106ca860cb08bb095a9bdbd130c2cb559ff191c1fa5c5651a816681784b16dd"}
Oct 31 01:57:01.064964 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:01.064517    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b106ca860cb08bb095a9bdbd130c2cb559ff191c1fa5c5651a816681784b16dd"
Oct 31 01:57:01.066972 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:01.066945    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" event={"ID":"eb25a85b-918d-4810-b90c-8b1d2e21a369","Type":"ContainerStarted","Data":"45fa3d2fd7a49585c76f0e1657f05eec53e79f285be333461c2c769b63917833"}
Oct 31 01:57:01.286128 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:01.286088    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:57:01.286555 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:01.286535    5293 patch_prober.go:29] interesting pod/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 31 01:57:01.286658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:01.286577    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 31 01:57:02.285946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:02.285911    5293 patch_prober.go:29] interesting pod/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 31 01:57:02.286481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:02.285981    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 31 01:57:03.285588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:03.285548    5293 patch_prober.go:29] interesting pod/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 31 01:57:03.285774 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:03.285618    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 31 01:57:04.286149 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:04.286118    5293 patch_prober.go:29] interesting pod/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 31 01:57:04.286552 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:04.286178    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 31 01:57:05.287066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:05.286673    5293 patch_prober.go:29] interesting pod/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 31 01:57:05.287066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:05.286747    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 31 01:57:06.013068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:06.007654    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:57:06.286346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:06.286263    5293 patch_prober.go:29] interesting pod/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 31 01:57:06.286346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:06.286325    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 31 01:57:07.164081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.164026    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"]
Oct 31 01:57:07.164811 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.164795    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a82e88a4-5e8e-473a-ab77-48050d2274a3" containerName="revoke-keys"
Oct 31 01:57:07.164856 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.164813    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82e88a4-5e8e-473a-ab77-48050d2274a3" containerName="revoke-keys"
Oct 31 01:57:07.164935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.164927    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82e88a4-5e8e-473a-ab77-48050d2274a3" containerName="revoke-keys"
Oct 31 01:57:07.168895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.168874    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.177099 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.177080    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae5a1c\"/\"kube-root-ca.crt\""
Oct 31 01:57:07.178698 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.178680    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae5a1c\"/\"openshift-service-ca.crt\""
Oct 31 01:57:07.179928 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.179907    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae5a1c\"/\"default-dockercfg-lnnkc\""
Oct 31 01:57:07.186733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.180176    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae5a1c\"/\"auth-test-model-ae5a1c-kserve-self-signed-certs\""
Oct 31 01:57:07.186849 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.186677    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"]
Oct 31 01:57:07.263220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.263185    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-dshm\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.263220 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.263224    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-model-cache\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.263445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.263323    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkvcz\" (UniqueName: \"kubernetes.io/projected/6a8d314b-9d5a-4174-b519-faf043158430-kube-api-access-qkvcz\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.263445 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.263407    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-tmp-dir\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.263534 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.263490    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8d314b-9d5a-4174-b519-faf043158430-tls-certs\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.263649 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.263628    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-home\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.289945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.289917    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:57:07.291116 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.291097    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:57:07.375067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.367320    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8d314b-9d5a-4174-b519-faf043158430-tls-certs\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.375067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.367526    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-home\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.375067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.367798    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-dshm\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.375067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.367838    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-model-cache\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.375067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.367904    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkvcz\" (UniqueName: \"kubernetes.io/projected/6a8d314b-9d5a-4174-b519-faf043158430-kube-api-access-qkvcz\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.375067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.367936    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-tmp-dir\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.375067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.368516    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-tmp-dir\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.375067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.370740    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-home\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.384066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.376364    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8d314b-9d5a-4174-b519-faf043158430-tls-certs\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.439248 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.430505    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-dshm\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.572078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.557785    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkvcz\" (UniqueName: \"kubernetes.io/projected/6a8d314b-9d5a-4174-b519-faf043158430-kube-api-access-qkvcz\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.610414 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.610366    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-model-cache\") pod \"auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") " pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:07.794658 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:07.789754    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:08.583291 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:08.583260    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"]
Oct 31 01:57:08.583754 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:57:08.583729    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a8d314b_9d5a_4174_b519_faf043158430.slice/crio-97e7aba6af32b788e5e78338eadbbf5c4c9fb99fc95c1805bd32ebabdce4016f WatchSource:0}: Error finding container 97e7aba6af32b788e5e78338eadbbf5c4c9fb99fc95c1805bd32ebabdce4016f: Status 404 returned error can't find the container with id 97e7aba6af32b788e5e78338eadbbf5c4c9fb99fc95c1805bd32ebabdce4016f
Oct 31 01:57:08.655257 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:08.655231    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:57:09.340672 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:09.340625    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr" event={"ID":"6a8d314b-9d5a-4174-b519-faf043158430","Type":"ContainerStarted","Data":"ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc"}
Oct 31 01:57:09.340882 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:09.340685    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr" event={"ID":"6a8d314b-9d5a-4174-b519-faf043158430","Type":"ContainerStarted","Data":"97e7aba6af32b788e5e78338eadbbf5c4c9fb99fc95c1805bd32ebabdce4016f"}
Oct 31 01:57:09.375186 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:09.370017    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr" podStartSLOduration=2.369999312 podStartE2EDuration="2.369999312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:09.362320815 +0000 UTC m=+2187.310969204" watchObservedRunningTime="2026-10-31 01:57:09.369999312 +0000 UTC m=+2187.318647705"
Oct 31 01:57:09.796090 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:09.792460    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:11.779017 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:57:11.778980    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:57:17.790636 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:17.790602    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:17.805526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:17.805501    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:18.510392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:18.510359    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 01:57:18.566656 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:18.566614    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"]
Oct 31 01:57:18.571389 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:18.571361    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c8b702-t4g5z"]
Oct 31 01:57:19.070062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:19.068224    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"]
Oct 31 01:57:19.076798 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:19.076739    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccddf611-chwqn"]
Oct 31 01:57:20.227388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:20.220726    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82e88a4-5e8e-473a-ab77-48050d2274a3" path="/var/lib/kubelet/pods/a82e88a4-5e8e-473a-ab77-48050d2274a3/volumes"
Oct 31 01:57:20.247067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:20.236634    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74868fb-1470-465e-b771-bc2a7d694a77" path="/var/lib/kubelet/pods/e74868fb-1470-465e-b771-bc2a7d694a77/volumes"
Oct 31 01:57:26.466384 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:26.466348    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"]
Oct 31 01:57:26.466889 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:26.466587    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy" containerID="cri-o://45fa3d2fd7a49585c76f0e1657f05eec53e79f285be333461c2c769b63917833" gracePeriod=30
Oct 31 01:57:31.941886 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:31.941842    5293 generic.go:356] "Generic (PLEG): container finished" podID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerID="45fa3d2fd7a49585c76f0e1657f05eec53e79f285be333461c2c769b63917833" exitCode=0
Oct 31 01:57:31.942454 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:31.942212    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" event={"ID":"eb25a85b-918d-4810-b90c-8b1d2e21a369","Type":"ContainerDied","Data":"45fa3d2fd7a49585c76f0e1657f05eec53e79f285be333461c2c769b63917833"}
Oct 31 01:57:31.942454 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:31.942319    5293 scope.go:122] "RemoveContainer" containerID="02637c1b02660536d09ff65f6307af3959ee8a1dd3c9faf5ee358b0a909480b2"
Oct 31 01:57:31.963302 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:31.963277    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:57:32.055599 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.055550    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-credential-socket\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.055800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.055675    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-certs\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.055800 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.055723    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-socket\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.055937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.055845    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-credential-socket" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:32.055937 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.055866    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-envoy\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.056103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.056026    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-certs" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:32.056172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.056149    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-socket" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:32.056329 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.056310    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-token\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-token\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.056411 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.056383    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-kube-api-access-tkhbh\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-kube-api-access-tkhbh\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.056471 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.056432    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-data\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.056531 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.056467    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-podinfo\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.056588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.056538    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb25a85b-918d-4810-b90c-8b1d2e21a369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb25a85b-918d-4810-b90c-8b1d2e21a369-istiod-ca-cert\") pod \"eb25a85b-918d-4810-b90c-8b1d2e21a369\" (UID: \"eb25a85b-918d-4810-b90c-8b1d2e21a369\") "
Oct 31 01:57:32.057182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.057154    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-data" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:32.057297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.057234    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb25a85b-918d-4810-b90c-8b1d2e21a369-istiod-ca-cert" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:32.057364 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.057350    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.057426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.057372    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.057426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.057388    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.057426 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.057404    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.058686 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.058648    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-envoy" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:32.061273 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.061247    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-token" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:32.061480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.061459    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-kube-api-access-tkhbh" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "kube-api-access-tkhbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:32.064249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.064222    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-podinfo" pod "eb25a85b-918d-4810-b90c-8b1d2e21a369" (UID: "eb25a85b-918d-4810-b90c-8b1d2e21a369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:57:32.162082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.160898    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.162082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.160941    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.162082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.160956    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhbh\" (UniqueName: \"kubernetes.io/projected/eb25a85b-918d-4810-b90c-8b1d2e21a369-kube-api-access-tkhbh\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.162082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.160971    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb25a85b-918d-4810-b90c-8b1d2e21a369-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.162082 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.160985    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb25a85b-918d-4810-b90c-8b1d2e21a369-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:32.588576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.588536    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"]
Oct 31 01:57:32.589695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.589663    5293 cpu_manager.go:414] "containerMap: removing container" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy"
Oct 31 01:57:32.589695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.589689    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy"
Oct 31 01:57:32.589930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.589730    5293 cpu_manager.go:414] "containerMap: removing container" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy"
Oct 31 01:57:32.589930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.589737    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy"
Oct 31 01:57:32.589930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.589919    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy"
Oct 31 01:57:32.590034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.589941    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" containerName="istio-proxy"
Oct 31 01:57:32.599337 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.599311    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.609093 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.605384    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-f353c2-openshift-default-dockercfg-2b9pk\""
Oct 31 01:57:32.622320 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.622291    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"]
Oct 31 01:57:32.680374 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680335    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-data\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.680596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680434    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69dr6\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-kube-api-access-69dr6\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.680596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680485    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-envoy\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.680596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680515    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-podinfo\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.680596 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680565    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-token\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.680858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680725    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-socket\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.680858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680852    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-certs\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.680980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680896    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istiod-ca-cert\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.680980 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.680965    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-credential-socket\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.797529    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-token\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.797746    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-socket\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.797881    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-certs\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.797930    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istiod-ca-cert\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.798008    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-credential-socket\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.798120    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-data\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.798211    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dr6\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-kube-api-access-69dr6\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.798263    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-envoy\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.801062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.798291    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-podinfo\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.853123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.802577    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istiod-ca-cert\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.853123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.803218    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-socket\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.853123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.803448    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-certs\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.853123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.803678    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-data\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.853123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.803895    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-credential-socket\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.853123 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.831297    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-envoy\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.884835 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.884748    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dr6\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-kube-api-access-69dr6\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.891079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.888570    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-token\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.891079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.889080    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-podinfo\") pod \"e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") " pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:32.944134 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:32.943831    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:33.174723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.170018    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm" event={"ID":"eb25a85b-918d-4810-b90c-8b1d2e21a369","Type":"ContainerDied","Data":"6d392cf2698516644ef2155b4139da93abbe1df702dc005a34b5b7fbac989549"}
Oct 31 01:57:33.174723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.170177    5293 scope.go:122] "RemoveContainer" containerID="45fa3d2fd7a49585c76f0e1657f05eec53e79f285be333461c2c769b63917833"
Oct 31 01:57:33.174723 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.170430    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"
Oct 31 01:57:33.381156 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.381126    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"]
Oct 31 01:57:33.388223 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.388191    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c8b702-openshift-default-58d85f4b45-6mpdm"]
Oct 31 01:57:33.486091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.486006    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"]
Oct 31 01:57:33.490231 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:57:33.490200    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd17cf8_95f3_46c7_848e_26f48e1a3751.slice/crio-eb405e9c18633cd67221586126654f06cd2f7fdd13ff2f0c63335c03db87f64e WatchSource:0}: Error finding container eb405e9c18633cd67221586126654f06cd2f7fdd13ff2f0c63335c03db87f64e: Status 404 returned error can't find the container with id eb405e9c18633cd67221586126654f06cd2f7fdd13ff2f0c63335c03db87f64e
Oct 31 01:57:33.492693 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.492662    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:57:33.492771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.492729    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:57:33.492771 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:33.492758    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:57:34.175367 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:34.175333    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb25a85b-918d-4810-b90c-8b1d2e21a369" path="/var/lib/kubelet/pods/eb25a85b-918d-4810-b90c-8b1d2e21a369/volumes"
Oct 31 01:57:34.180970 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:34.180936    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" event={"ID":"3bd17cf8-95f3-46c7-848e-26f48e1a3751","Type":"ContainerStarted","Data":"f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0"}
Oct 31 01:57:34.181159 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:34.180994    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" event={"ID":"3bd17cf8-95f3-46c7-848e-26f48e1a3751","Type":"ContainerStarted","Data":"eb405e9c18633cd67221586126654f06cd2f7fdd13ff2f0c63335c03db87f64e"}
Oct 31 01:57:34.209075 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:34.208989    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" podStartSLOduration=2.208973143 podStartE2EDuration="2.208973143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:34.201978401 +0000 UTC m=+2212.150626794" watchObservedRunningTime="2026-10-31 01:57:34.208973143 +0000 UTC m=+2212.157621536"
Oct 31 01:57:34.946016 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:34.945984    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:34.952544 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:34.952521    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:35.031676 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:35.031511    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"]
Oct 31 01:57:35.036314 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:35.036291    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c678dcc-8rgx5"]
Oct 31 01:57:35.190392 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:35.190359    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:35.191262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:35.191244    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:57:36.181642 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:36.181603    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50" path="/var/lib/kubelet/pods/88e5e07d-1d4c-4b2f-8ebe-ae787f07fa50/volumes"
Oct 31 01:57:37.925460 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:37.925332    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"]
Oct 31 01:57:37.937142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:37.937123    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:37.937369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:37.937290    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"]
Oct 31 01:57:37.944158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:37.944126    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"]
Oct 31 01:57:37.946336 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:37.946314    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f353c2\""
Oct 31 01:57:37.975206 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:37.975178    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"]
Oct 31 01:57:37.975378 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:37.975183    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:37.980130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:37.979996    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-f353c2-dockercfg-bzzxj\""
Oct 31 01:57:38.118169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.112185    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a166c046-5d6e-4725-83b0-3e7ffe155be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8\" (UID: \"a166c046-5d6e-4725-83b0-3e7ffe155be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:38.118169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.112262    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0b749c-921d-473d-98fd-c05c70279567-plugins-config-volume\") pod \"payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw\" (UID: \"4e0b749c-921d-473d-98fd-c05c70279567\") " pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:38.118169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.112359    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkjc\" (UniqueName: \"kubernetes.io/projected/4e0b749c-921d-473d-98fd-c05c70279567-kube-api-access-bvkjc\") pod \"payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw\" (UID: \"4e0b749c-921d-473d-98fd-c05c70279567\") " pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:38.118169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.112406    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xctw\" (UniqueName: \"kubernetes.io/projected/a166c046-5d6e-4725-83b0-3e7ffe155be3-kube-api-access-4xctw\") pod \"payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8\" (UID: \"a166c046-5d6e-4725-83b0-3e7ffe155be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:38.219947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.216025    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a166c046-5d6e-4725-83b0-3e7ffe155be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8\" (UID: \"a166c046-5d6e-4725-83b0-3e7ffe155be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:38.219947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.216136    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0b749c-921d-473d-98fd-c05c70279567-plugins-config-volume\") pod \"payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw\" (UID: \"4e0b749c-921d-473d-98fd-c05c70279567\") " pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:38.219947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.216252    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkjc\" (UniqueName: \"kubernetes.io/projected/4e0b749c-921d-473d-98fd-c05c70279567-kube-api-access-bvkjc\") pod \"payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw\" (UID: \"4e0b749c-921d-473d-98fd-c05c70279567\") " pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:38.219947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.216306    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xctw\" (UniqueName: \"kubernetes.io/projected/a166c046-5d6e-4725-83b0-3e7ffe155be3-kube-api-access-4xctw\") pod \"payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8\" (UID: \"a166c046-5d6e-4725-83b0-3e7ffe155be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:38.219947 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.217786    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a166c046-5d6e-4725-83b0-3e7ffe155be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8\" (UID: \"a166c046-5d6e-4725-83b0-3e7ffe155be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:38.229309 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.226601    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0b749c-921d-473d-98fd-c05c70279567-plugins-config-volume\") pod \"payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw\" (UID: \"4e0b749c-921d-473d-98fd-c05c70279567\") " pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:38.259067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.248880    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xctw\" (UniqueName: \"kubernetes.io/projected/a166c046-5d6e-4725-83b0-3e7ffe155be3-kube-api-access-4xctw\") pod \"payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8\" (UID: \"a166c046-5d6e-4725-83b0-3e7ffe155be3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:38.294078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.289999    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkjc\" (UniqueName: \"kubernetes.io/projected/4e0b749c-921d-473d-98fd-c05c70279567-kube-api-access-bvkjc\") pod \"payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw\" (UID: \"4e0b749c-921d-473d-98fd-c05c70279567\") " pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:38.294078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.291179    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:38.583114 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.582749    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:38.722917 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.722890    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"]
Oct 31 01:57:38.725530 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:57:38.725452    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda166c046_5d6e_4725_83b0_3e7ffe155be3.slice/crio-e98812e8b5920b0f4bdff171eafc3a3406fca8a5210e3de596edfea9d2d780ed WatchSource:0}: Error finding container e98812e8b5920b0f4bdff171eafc3a3406fca8a5210e3de596edfea9d2d780ed: Status 404 returned error can't find the container with id e98812e8b5920b0f4bdff171eafc3a3406fca8a5210e3de596edfea9d2d780ed
Oct 31 01:57:38.956015 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:38.955980    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"]
Oct 31 01:57:38.956416 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:57:38.955973    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0b749c_921d_473d_98fd_c05c70279567.slice/crio-e80fdac39c58e6c3187857d9476ec164bfb6875abe78616badbda4d21eeb6b61 WatchSource:0}: Error finding container e80fdac39c58e6c3187857d9476ec164bfb6875abe78616badbda4d21eeb6b61: Status 404 returned error can't find the container with id e80fdac39c58e6c3187857d9476ec164bfb6875abe78616badbda4d21eeb6b61
Oct 31 01:57:39.373136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:39.368193    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8" event={"ID":"a166c046-5d6e-4725-83b0-3e7ffe155be3","Type":"ContainerStarted","Data":"489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9"}
Oct 31 01:57:39.373136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:39.368258    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:39.373136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:39.368282    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8" event={"ID":"a166c046-5d6e-4725-83b0-3e7ffe155be3","Type":"ContainerStarted","Data":"e98812e8b5920b0f4bdff171eafc3a3406fca8a5210e3de596edfea9d2d780ed"}
Oct 31 01:57:39.373136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:39.370206    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw" event={"ID":"4e0b749c-921d-473d-98fd-c05c70279567","Type":"ContainerStarted","Data":"df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02"}
Oct 31 01:57:39.373136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:39.370257    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw" event={"ID":"4e0b749c-921d-473d-98fd-c05c70279567","Type":"ContainerStarted","Data":"e80fdac39c58e6c3187857d9476ec164bfb6875abe78616badbda4d21eeb6b61"}
Oct 31 01:57:39.373136 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:39.370372    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:57:39.387461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:39.387409    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8" podStartSLOduration=2.38739405 podStartE2EDuration="2.38739405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:39.385317593 +0000 UTC m=+2217.333965985" watchObservedRunningTime="2026-10-31 01:57:39.38739405 +0000 UTC m=+2217.336042442"
Oct 31 01:57:39.406846 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:39.406791    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw" podStartSLOduration=2.406776792 podStartE2EDuration="2.406776792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:39.402416582 +0000 UTC m=+2217.351064974" watchObservedRunningTime="2026-10-31 01:57:39.406776792 +0000 UTC m=+2217.355425181"
Oct 31 01:57:41.809148 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:57:41.805188    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:57:45.783712 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:45.783680    5293 scope.go:122] "RemoveContainer" containerID="9547cf09b17016bf08b222c70cf779ebedd5e7f7e07ccd83e6d0781051cc2bab"
Oct 31 01:57:45.795270 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:45.795248    5293 scope.go:122] "RemoveContainer" containerID="a8fc619eb86828af1858d3cc7842cbb6bfa063faf96f71d497602f872fe45377"
Oct 31 01:57:45.805919 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:45.805895    5293 scope.go:122] "RemoveContainer" containerID="75f9a3ab7a0a045d3b00d0175315e1b2a730acc4e5b43f8a40fc0c16c38d5c59"
Oct 31 01:57:45.817775 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:45.817756    5293 scope.go:122] "RemoveContainer" containerID="77aa574571ae3d9b68db331ffb4804680adff6bd17b04555f1ad78aa95fdd8d1"
Oct 31 01:57:45.828680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:45.828652    5293 scope.go:122] "RemoveContainer" containerID="3df569da3626e2c81452ab5ad92a4ac258fa1f5ebe927b3fa9a898d24c3169a3"
Oct 31 01:57:48.045668 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:48.045624    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"]
Oct 31 01:57:48.055985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:48.055953    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212cd153-92fwz"]
Oct 31 01:57:48.175522 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:48.175487    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90b9e1a-8690-40a4-8279-49c974f5b666" path="/var/lib/kubelet/pods/c90b9e1a-8690-40a4-8279-49c974f5b666/volumes"
Oct 31 01:57:50.416290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:50.416260    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:57:50.417474 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:57:50.416346    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:58:03.030346 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:03.030289    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"]
Oct 31 01:58:03.036536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:03.036507    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e7b37a-rgsbf"]
Oct 31 01:58:04.247178 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:04.247137    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06" path="/var/lib/kubelet/pods/f1fcdfcb-af8d-4dc5-80e6-46d7e8b05a06/volumes"
Oct 31 01:58:05.929158 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:05.929116    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"]
Oct 31 01:58:05.934353 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:05.934330    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:05.938436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:05.938415    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae5a1c-openshift-default-dockercfg-ctkcr\""
Oct 31 01:58:05.947699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:05.947680    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"]
Oct 31 01:58:06.090773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.090740    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.090968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.090872    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.090968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.090925    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.090968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.090951    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.091138 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.091107    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.091184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.091163    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.091244 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.091231    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.091298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.091282    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.091375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.091354    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwn7\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-kube-api-access-4pwn7\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193297 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193225    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193307    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193358    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193378    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193428    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193480 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193449    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193533    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193605    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193635    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwn7\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-kube-api-access-4pwn7\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.193765 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193660    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.194005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193772    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.194005 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.193868    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.194196 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.194177    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.194447 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.194414    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.195901 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.195880    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.196112 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.196092    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.205472 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.205451    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.205684 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.205662    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwn7\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-kube-api-access-4pwn7\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.255831 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.255540    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:06.655410 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:58:06.655374    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8baa7adb_dc5a_4f8c_b24a_ced615febf3b.slice/crio-c63f71587b7e2f779dddf5a26ccbd1f25bab0eaaa836e1df42d10c9d712fa630 WatchSource:0}: Error finding container c63f71587b7e2f779dddf5a26ccbd1f25bab0eaaa836e1df42d10c9d712fa630: Status 404 returned error can't find the container with id c63f71587b7e2f779dddf5a26ccbd1f25bab0eaaa836e1df42d10c9d712fa630
Oct 31 01:58:06.656103 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.656077    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"]
Oct 31 01:58:06.658267 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.658234    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:06.658334 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.658311    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:06.658371 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:06.658346    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:07.107588 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:07.102483    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" event={"ID":"8baa7adb-dc5a-4f8c-b24a-ced615febf3b","Type":"ContainerStarted","Data":"27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6"}
Oct 31 01:58:07.108264 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:07.108237    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" event={"ID":"8baa7adb-dc5a-4f8c-b24a-ced615febf3b","Type":"ContainerStarted","Data":"c63f71587b7e2f779dddf5a26ccbd1f25bab0eaaa836e1df42d10c9d712fa630"}
Oct 31 01:58:07.134825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:07.134764    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podStartSLOduration=2.134748831 podStartE2EDuration="2.134748831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:07.132698845 +0000 UTC m=+2245.081347236" watchObservedRunningTime="2026-10-31 01:58:07.134748831 +0000 UTC m=+2245.083397227"
Oct 31 01:58:07.256450 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:07.256414    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:07.257792 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:07.257769    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:58:07.257856 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:07.257833    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:58:08.256687 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:08.256653    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:58:08.257127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:08.256715    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:58:09.257498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:09.257463    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:58:09.258217 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:09.258183    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:58:10.257154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:10.257112    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:58:10.257381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:10.257178    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:58:11.256893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:11.256853    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:58:11.257338 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:11.256915    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:58:11.779253 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:58:11.779211    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:58:12.136141 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:12.136023    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:58:12.261946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:12.261444    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:58:12.261946 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:12.261526    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:58:13.262174 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:13.262139    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:58:13.262778 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:13.262746    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:58:14.097194 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:14.097134    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"]
Oct 31 01:58:14.140063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:14.127325    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc6cc10-6l6dj"]
Oct 31 01:58:14.220983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:14.220946    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3084e146-3fa1-4cca-9221-2d02f59bf2e5" path="/var/lib/kubelet/pods/3084e146-3fa1-4cca-9221-2d02f59bf2e5/volumes"
Oct 31 01:58:14.282647 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:14.282612    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 31 01:58:14.283422 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:14.283385    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 31 01:58:15.261958 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:15.261924    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:15.262481 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:15.262459    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:15.263236 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:15.263216    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 01:58:19.574171 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.567128    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"]
Oct 31 01:58:19.596945 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.596912    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.606485 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.606453    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:58:19.607456 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.607431    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:58:19.645923 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.645894    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"]
Oct 31 01:58:19.650380 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.650347    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:58:19.655269 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.655243    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:58:19.768895 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.768742    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.769335 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.769307    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jx8\" (UniqueName: \"kubernetes.io/projected/2d5270b8-31f3-4aa3-8e79-2d461058ea37-kube-api-access-w8jx8\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.769680 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.769646    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.871121 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.871009    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.871518 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.871499    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jx8\" (UniqueName: \"kubernetes.io/projected/2d5270b8-31f3-4aa3-8e79-2d461058ea37-kube-api-access-w8jx8\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.871773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.871758    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.873078 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.873034    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.884402 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.884373    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:19.886875 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:19.886849    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jx8\" (UniqueName: \"kubernetes.io/projected/2d5270b8-31f3-4aa3-8e79-2d461058ea37-kube-api-access-w8jx8\") pod \"maas-api-revoke-keys-e2e-autores-f353c2-qvs87\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:20.033469 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:20.033437    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:20.353120 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:20.353090    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"]
Oct 31 01:58:20.354342 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:58:20.354312    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5270b8_31f3_4aa3_8e79_2d461058ea37.slice/crio-153afb6396b9bc731a745a3b00012379513ff4977f81aecc8769a339b3919e79 WatchSource:0}: Error finding container 153afb6396b9bc731a745a3b00012379513ff4977f81aecc8769a339b3919e79: Status 404 returned error can't find the container with id 153afb6396b9bc731a745a3b00012379513ff4977f81aecc8769a339b3919e79
Oct 31 01:58:20.379315 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:20.379284    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87" event={"ID":"2d5270b8-31f3-4aa3-8e79-2d461058ea37","Type":"ContainerStarted","Data":"153afb6396b9bc731a745a3b00012379513ff4977f81aecc8769a339b3919e79"}
Oct 31 01:58:21.025694 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:21.025607    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"]
Oct 31 01:58:21.029902 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:21.029876    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0a58-t4skz"]
Oct 31 01:58:21.389730 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:21.389642    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87" event={"ID":"2d5270b8-31f3-4aa3-8e79-2d461058ea37","Type":"ContainerStarted","Data":"1864c4be79b0c17ca6e7e4332f44275af54e14523509a40561f1d366a09621da"}
Oct 31 01:58:21.412278 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:21.412226    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87" podStartSLOduration=2.412211137 podStartE2EDuration="2.412211137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:21.406351028 +0000 UTC m=+2259.354999433" watchObservedRunningTime="2026-10-31 01:58:21.412211137 +0000 UTC m=+2259.360859535"
Oct 31 01:58:22.177154 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:22.177075    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06b6bc6-0460-4d28-85d5-eacbbe50a06c" path="/var/lib/kubelet/pods/d06b6bc6-0460-4d28-85d5-eacbbe50a06c/volumes"
Oct 31 01:58:22.401650 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:22.401603    5293 generic.go:356] "Generic (PLEG): container finished" podID="2d5270b8-31f3-4aa3-8e79-2d461058ea37" containerID="1864c4be79b0c17ca6e7e4332f44275af54e14523509a40561f1d366a09621da" exitCode=0
Oct 31 01:58:22.401907 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:22.401682    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87" event={"ID":"2d5270b8-31f3-4aa3-8e79-2d461058ea37","Type":"ContainerDied","Data":"1864c4be79b0c17ca6e7e4332f44275af54e14523509a40561f1d366a09621da"}
Oct 31 01:58:23.635909 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.635878    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:23.709643 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.709561    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-xks-ca\") pod \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") "
Oct 31 01:58:23.709824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.709678    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5270b8-31f3-4aa3-8e79-2d461058ea37-kube-api-access-w8jx8\" (UniqueName: \"kubernetes.io/projected/2d5270b8-31f3-4aa3-8e79-2d461058ea37-kube-api-access-w8jx8\") pod \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") "
Oct 31 01:58:23.710030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.709994    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-openshift-ca\") pod \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\" (UID: \"2d5270b8-31f3-4aa3-8e79-2d461058ea37\") "
Oct 31 01:58:23.714172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.712562    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-xks-ca" pod "2d5270b8-31f3-4aa3-8e79-2d461058ea37" (UID: "2d5270b8-31f3-4aa3-8e79-2d461058ea37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:58:23.714172 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.712659    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-openshift-ca" pod "2d5270b8-31f3-4aa3-8e79-2d461058ea37" (UID: "2d5270b8-31f3-4aa3-8e79-2d461058ea37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:58:23.714567 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.714541    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5270b8-31f3-4aa3-8e79-2d461058ea37-kube-api-access-w8jx8" pod "2d5270b8-31f3-4aa3-8e79-2d461058ea37" (UID: "2d5270b8-31f3-4aa3-8e79-2d461058ea37"). InnerVolumeSpecName "kube-api-access-w8jx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:23.714796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.714732    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:23.714796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.714748    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5270b8-31f3-4aa3-8e79-2d461058ea37-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:23.714796 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:23.714757    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jx8\" (UniqueName: \"kubernetes.io/projected/2d5270b8-31f3-4aa3-8e79-2d461058ea37-kube-api-access-w8jx8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:24.030965 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:24.030881    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"]
Oct 31 01:58:24.034922 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:24.034891    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e73624a-k8jbw"]
Oct 31 01:58:24.190157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:24.190127    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97dc99e-51dd-4ede-9b0d-9b7595b7d200" path="/var/lib/kubelet/pods/e97dc99e-51dd-4ede-9b0d-9b7595b7d200/volumes"
Oct 31 01:58:24.458214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:24.458185    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"
Oct 31 01:58:24.458214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:24.458192    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87" event={"ID":"2d5270b8-31f3-4aa3-8e79-2d461058ea37","Type":"ContainerDied","Data":"153afb6396b9bc731a745a3b00012379513ff4977f81aecc8769a339b3919e79"}
Oct 31 01:58:24.458444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:24.458251    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="153afb6396b9bc731a745a3b00012379513ff4977f81aecc8769a339b3919e79"
Oct 31 01:58:26.476984 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:26.476950    5293 generic.go:356] "Generic (PLEG): container finished" podID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerID="f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0" exitCode=0
Oct 31 01:58:26.477442 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:26.477074    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" event={"ID":"3bd17cf8-95f3-46c7-848e-26f48e1a3751","Type":"ContainerDied","Data":"f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0"}
Oct 31 01:58:26.477619 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:26.477604    5293 scope.go:122] "RemoveContainer" containerID="f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0"
Oct 31 01:58:26.479415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:26.479384    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:26.479489 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:26.479461    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:26.479527 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:26.479493    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:27.487571 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:27.487543    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" event={"ID":"3bd17cf8-95f3-46c7-848e-26f48e1a3751","Type":"ContainerStarted","Data":"4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356"}
Oct 31 01:58:27.945718 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:27.945680    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:58:27.946003 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:27.945980    5293 patch_prober.go:29] interesting pod/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 31 01:58:27.946111 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:27.946053    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 31 01:58:28.945432 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:28.945394    5293 patch_prober.go:29] interesting pod/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 31 01:58:28.945825 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:28.945473    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 31 01:58:29.684777 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:29.684744    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"]
Oct 31 01:58:29.723064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:29.718685    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"]
Oct 31 01:58:29.945783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:29.945670    5293 patch_prober.go:29] interesting pod/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 31 01:58:29.945783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:29.945729    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 31 01:58:30.948198 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:30.948155    5293 patch_prober.go:29] interesting pod/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 31 01:58:30.948887 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:30.948845    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 31 01:58:31.103603 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:31.103556    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"]
Oct 31 01:58:31.372521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:31.347012    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f353c2-qvs87"]
Oct 31 01:58:31.954184 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:31.954149    5293 patch_prober.go:29] interesting pod/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 31 01:58:31.954843 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:31.954808    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 31 01:58:32.223516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:32.223420    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5270b8-31f3-4aa3-8e79-2d461058ea37" path="/var/lib/kubelet/pods/2d5270b8-31f3-4aa3-8e79-2d461058ea37/volumes"
Oct 31 01:58:32.945826 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:32.945781    5293 patch_prober.go:29] interesting pod/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 31 01:58:32.946079 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:32.945892    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 31 01:58:34.253562 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:34.253520    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:58:34.254153 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:34.253924    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:58:34.255707 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:34.255684    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:58:34.685157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:34.685123    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw" podUID="4e0b749c-921d-473d-98fd-c05c70279567" containerName="payload-processing" containerID="cri-o://df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02" gracePeriod=25
Oct 31 01:58:34.719877 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:34.719824    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8" podUID="a166c046-5d6e-4725-83b0-3e7ffe155be3" containerName="payload-pre-processing" containerID="cri-o://489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9" gracePeriod=25
Oct 31 01:58:35.062239 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.062211    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:58:35.103881 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.103851    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:58:35.161911 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.161875    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0b749c-921d-473d-98fd-c05c70279567-kube-api-access-bvkjc\" (UniqueName: \"kubernetes.io/projected/4e0b749c-921d-473d-98fd-c05c70279567-kube-api-access-bvkjc\") pod \"4e0b749c-921d-473d-98fd-c05c70279567\" (UID: \"4e0b749c-921d-473d-98fd-c05c70279567\") "
Oct 31 01:58:35.162182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.161951    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a166c046-5d6e-4725-83b0-3e7ffe155be3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a166c046-5d6e-4725-83b0-3e7ffe155be3-plugins-config-volume\") pod \"a166c046-5d6e-4725-83b0-3e7ffe155be3\" (UID: \"a166c046-5d6e-4725-83b0-3e7ffe155be3\") "
Oct 31 01:58:35.162182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.162064    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a166c046-5d6e-4725-83b0-3e7ffe155be3-kube-api-access-4xctw\" (UniqueName: \"kubernetes.io/projected/a166c046-5d6e-4725-83b0-3e7ffe155be3-kube-api-access-4xctw\") pod \"a166c046-5d6e-4725-83b0-3e7ffe155be3\" (UID: \"a166c046-5d6e-4725-83b0-3e7ffe155be3\") "
Oct 31 01:58:35.162182 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.162158    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e0b749c-921d-473d-98fd-c05c70279567-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0b749c-921d-473d-98fd-c05c70279567-plugins-config-volume\") pod \"4e0b749c-921d-473d-98fd-c05c70279567\" (UID: \"4e0b749c-921d-473d-98fd-c05c70279567\") "
Oct 31 01:58:35.162503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.162470    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a166c046-5d6e-4725-83b0-3e7ffe155be3-plugins-config-volume" pod "a166c046-5d6e-4725-83b0-3e7ffe155be3" (UID: "a166c046-5d6e-4725-83b0-3e7ffe155be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:58:35.162593 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.162522    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0b749c-921d-473d-98fd-c05c70279567-plugins-config-volume" pod "4e0b749c-921d-473d-98fd-c05c70279567" (UID: "4e0b749c-921d-473d-98fd-c05c70279567"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:58:35.162968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.162911    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a166c046-5d6e-4725-83b0-3e7ffe155be3-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:35.162968 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.162938    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0b749c-921d-473d-98fd-c05c70279567-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:35.165219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.165198    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0b749c-921d-473d-98fd-c05c70279567-kube-api-access-bvkjc" pod "4e0b749c-921d-473d-98fd-c05c70279567" (UID: "4e0b749c-921d-473d-98fd-c05c70279567"). InnerVolumeSpecName "kube-api-access-bvkjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:35.165354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.165336    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a166c046-5d6e-4725-83b0-3e7ffe155be3-kube-api-access-4xctw" pod "a166c046-5d6e-4725-83b0-3e7ffe155be3" (UID: "a166c046-5d6e-4725-83b0-3e7ffe155be3"). InnerVolumeSpecName "kube-api-access-4xctw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:35.265424 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.265373    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkjc\" (UniqueName: \"kubernetes.io/projected/4e0b749c-921d-473d-98fd-c05c70279567-kube-api-access-bvkjc\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:35.265892 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.265495    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xctw\" (UniqueName: \"kubernetes.io/projected/a166c046-5d6e-4725-83b0-3e7ffe155be3-kube-api-access-4xctw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:35.698896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.698853    5293 generic.go:356] "Generic (PLEG): container finished" podID="a166c046-5d6e-4725-83b0-3e7ffe155be3" containerID="489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9" exitCode=0
Oct 31 01:58:35.699168 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.699031    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8" event={"ID":"a166c046-5d6e-4725-83b0-3e7ffe155be3","Type":"ContainerDied","Data":"489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9"}
Oct 31 01:58:35.699275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.699216    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8" event={"ID":"a166c046-5d6e-4725-83b0-3e7ffe155be3","Type":"ContainerDied","Data":"e98812e8b5920b0f4bdff171eafc3a3406fca8a5210e3de596edfea9d2d780ed"}
Oct 31 01:58:35.699275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.699250    5293 scope.go:122] "RemoveContainer" containerID="489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9"
Oct 31 01:58:35.699456 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.699437    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"
Oct 31 01:58:35.705977 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.705956    5293 generic.go:356] "Generic (PLEG): container finished" podID="4e0b749c-921d-473d-98fd-c05c70279567" containerID="df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02" exitCode=0
Oct 31 01:58:35.707457 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.707441    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"
Oct 31 01:58:35.713214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.713026    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw" event={"ID":"4e0b749c-921d-473d-98fd-c05c70279567","Type":"ContainerDied","Data":"df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02"}
Oct 31 01:58:35.713214 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.713109    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw" event={"ID":"4e0b749c-921d-473d-98fd-c05c70279567","Type":"ContainerDied","Data":"e80fdac39c58e6c3187857d9476ec164bfb6875abe78616badbda4d21eeb6b61"}
Oct 31 01:58:35.729157 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.729131    5293 scope.go:122] "RemoveContainer" containerID="489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9"
Oct 31 01:58:35.729659 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:58:35.729633    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9\": container with ID starting with 489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9 not found: ID does not exist" containerID="489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9"
Oct 31 01:58:35.729773 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.729742    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9"} err="failed to get container status \"489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9\": rpc error: code = NotFound desc = could not find container \"489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9\": container with ID starting with 489ccff6712052e9accdbe264a18552729066b104c2e313cb4e6a68754ed4ff9 not found: ID does not exist"
Oct 31 01:58:35.729848 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.729775    5293 scope.go:122] "RemoveContainer" containerID="df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02"
Oct 31 01:58:35.744995 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.744975    5293 scope.go:122] "RemoveContainer" containerID="df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02"
Oct 31 01:58:35.745413 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:58:35.745392    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02\": container with ID starting with df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02 not found: ID does not exist" containerID="df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02"
Oct 31 01:58:35.745516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.745421    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02"} err="failed to get container status \"df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02\": rpc error: code = NotFound desc = could not find container \"df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02\": container with ID starting with df9569b4b8d7ee8b44d24b145f0c2816384ceef4cc612b89b33b81a330917b02 not found: ID does not exist"
Oct 31 01:58:35.782404 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.782382    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"]
Oct 31 01:58:35.814597 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.814512    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f353c2-598dd8f8d5-vx9kw"]
Oct 31 01:58:35.871283 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.871227    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"]
Oct 31 01:58:35.915618 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:35.915586    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f353c2-57fcc4666c-4jdp8"]
Oct 31 01:58:36.234064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:36.231270    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e0b749c-921d-473d-98fd-c05c70279567" path="/var/lib/kubelet/pods/4e0b749c-921d-473d-98fd-c05c70279567/volumes"
Oct 31 01:58:36.234064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:36.232016    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a166c046-5d6e-4725-83b0-3e7ffe155be3" path="/var/lib/kubelet/pods/a166c046-5d6e-4725-83b0-3e7ffe155be3/volumes"
Oct 31 01:58:37.065560 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:37.065526    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"]
Oct 31 01:58:37.075935 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:37.075895    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3bf7b9-m92h8"]
Oct 31 01:58:38.245444 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:38.240174    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1948dd27-a949-499c-a849-1cee19c64633" path="/var/lib/kubelet/pods/1948dd27-a949-499c-a849-1cee19c64633/volumes"
Oct 31 01:58:39.117701 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:39.117653    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"]
Oct 31 01:58:39.118354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:39.118317    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy" containerID="cri-o://4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356" gracePeriod=30
Oct 31 01:58:41.781827 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:58:41.781790    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:58:44.441898 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.441874    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:58:44.619861 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.619828    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-token\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-token\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.619896    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-certs\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.619925    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-kube-api-access-69dr6\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-kube-api-access-69dr6\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.619945    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-envoy\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620094 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620084    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istiod-ca-cert\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620117    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-podinfo\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620253    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-socket\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620287    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-data\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620354 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620312    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-certs" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:44.620569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620488    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-socket" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:44.620569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620517    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-data" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:44.620569 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620561    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-credential-socket\") pod \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\" (UID: \"3bd17cf8-95f3-46c7-848e-26f48e1a3751\") "
Oct 31 01:58:44.620900 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.620801    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-credential-socket" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:44.621249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.621220    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istiod-ca-cert" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:58:44.621358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.621307    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.621358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.621333    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.621358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.621350    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.621526 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.621369    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.622561 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.622538    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-token" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:44.623132 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.623104    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-podinfo" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:58:44.623523 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.623504    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-envoy" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:44.623608 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.623519    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-kube-api-access-69dr6" pod "3bd17cf8-95f3-46c7-848e-26f48e1a3751" (UID: "3bd17cf8-95f3-46c7-848e-26f48e1a3751"). InnerVolumeSpecName "kube-api-access-69dr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:44.722525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.722429    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.722525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.722456    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.722525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.722465    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.722525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.722473    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69dr6\" (UniqueName: \"kubernetes.io/projected/3bd17cf8-95f3-46c7-848e-26f48e1a3751-kube-api-access-69dr6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.722525 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.722484    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd17cf8-95f3-46c7-848e-26f48e1a3751-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.800362 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.800327    5293 generic.go:356] "Generic (PLEG): container finished" podID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerID="4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356" exitCode=0
Oct 31 01:58:44.800565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.800383    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" event={"ID":"3bd17cf8-95f3-46c7-848e-26f48e1a3751","Type":"ContainerDied","Data":"4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356"}
Oct 31 01:58:44.800565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.800410    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"
Oct 31 01:58:44.800565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.800453    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq" event={"ID":"3bd17cf8-95f3-46c7-848e-26f48e1a3751","Type":"ContainerDied","Data":"eb405e9c18633cd67221586126654f06cd2f7fdd13ff2f0c63335c03db87f64e"}
Oct 31 01:58:44.800565 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.800475    5293 scope.go:122] "RemoveContainer" containerID="4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356"
Oct 31 01:58:44.813819 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.813793    5293 scope.go:122] "RemoveContainer" containerID="f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0"
Oct 31 01:58:44.826740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.824362    5293 scope.go:122] "RemoveContainer" containerID="4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356"
Oct 31 01:58:44.826740 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:58:44.824694    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356\": container with ID starting with 4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356 not found: ID does not exist" containerID="4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356"
Oct 31 01:58:44.826740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.824730    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356"} err="failed to get container status \"4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356\": rpc error: code = NotFound desc = could not find container \"4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356\": container with ID starting with 4dd142d186a11999e44d1ec6e766de5a13736aad6ecaf92f4c89ca157d032356 not found: ID does not exist"
Oct 31 01:58:44.826740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.824755    5293 scope.go:122] "RemoveContainer" containerID="f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0"
Oct 31 01:58:44.826740 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:58:44.825244    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0\": container with ID starting with f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0 not found: ID does not exist" containerID="f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0"
Oct 31 01:58:44.826740 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.825269    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0"} err="failed to get container status \"f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0\": rpc error: code = NotFound desc = could not find container \"f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0\": container with ID starting with f904161ff9975e6d8496001b7579d03ee55fc5c53d46b95aba6f052e40015cf0 not found: ID does not exist"
Oct 31 01:58:44.834056 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.830371    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"]
Oct 31 01:58:44.836366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:44.836340    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-f353c2-openshift-default-65ff77f687-2wmmq"]
Oct 31 01:58:45.027249 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:45.027166    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"]
Oct 31 01:58:45.034253 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:45.034215    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f40888-8mqzq"]
Oct 31 01:58:45.906794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:45.906704    5293 scope.go:122] "RemoveContainer" containerID="6525c8fedfa986088a6806e97312e9bed809725794b4611f4fac4da7d81fb715"
Oct 31 01:58:45.919733 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:45.919705    5293 scope.go:122] "RemoveContainer" containerID="0b04f3f5a4885ac6da01b00da7ac52ad8787798a0614148c6578c136fb2ed792"
Oct 31 01:58:45.931422 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:45.931401    5293 scope.go:122] "RemoveContainer" containerID="d355cbc0cd448279bd563e64950e385249d75e354c634eb849ecb36382ea0a69"
Oct 31 01:58:45.941671 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:45.941651    5293 scope.go:122] "RemoveContainer" containerID="4934ac4d24165d5e771b60298caa80a08b56a3defba2145e375b28eecf06d7b6"
Oct 31 01:58:45.951524 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:45.951504    5293 scope.go:122] "RemoveContainer" containerID="63661078ffaf934299db38c08232c91d2566a39687590edea44c6d36ef8a1a24"
Oct 31 01:58:45.961366 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:45.961347    5293 scope.go:122] "RemoveContainer" containerID="5d8a568a8c7146a5ed892d13f0aa6d4dd8e5e9a25b4c2e458c0cc9c23af2e875"
Oct 31 01:58:46.174595 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:46.174519    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" path="/var/lib/kubelet/pods/3bd17cf8-95f3-46c7-848e-26f48e1a3751/volumes"
Oct 31 01:58:46.175013 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:46.174999    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f98a85-24fd-481a-b607-df3260ae9db8" path="/var/lib/kubelet/pods/94f98a85-24fd-481a-b607-df3260ae9db8/volumes"
Oct 31 01:58:51.945358 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.945321    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"]
Oct 31 01:58:51.946461 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946439    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a166c046-5d6e-4725-83b0-3e7ffe155be3" containerName="payload-pre-processing"
Oct 31 01:58:51.946536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946464    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a166c046-5d6e-4725-83b0-3e7ffe155be3" containerName="payload-pre-processing"
Oct 31 01:58:51.946536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946494    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy"
Oct 31 01:58:51.946536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946501    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy"
Oct 31 01:58:51.946536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946516    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy"
Oct 31 01:58:51.946536 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946524    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy"
Oct 31 01:58:51.946702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946554    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2d5270b8-31f3-4aa3-8e79-2d461058ea37" containerName="revoke-keys"
Oct 31 01:58:51.946702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946561    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5270b8-31f3-4aa3-8e79-2d461058ea37" containerName="revoke-keys"
Oct 31 01:58:51.946702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946584    5293 cpu_manager.go:414] "containerMap: removing container" podUID="4e0b749c-921d-473d-98fd-c05c70279567" containerName="payload-processing"
Oct 31 01:58:51.946702 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946591    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0b749c-921d-473d-98fd-c05c70279567" containerName="payload-processing"
Oct 31 01:58:51.946827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946736    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a166c046-5d6e-4725-83b0-3e7ffe155be3" containerName="payload-pre-processing"
Oct 31 01:58:51.946827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946750    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy"
Oct 31 01:58:51.946827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946762    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd17cf8-95f3-46c7-848e-26f48e1a3751" containerName="istio-proxy"
Oct 31 01:58:51.946827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946775    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5270b8-31f3-4aa3-8e79-2d461058ea37" containerName="revoke-keys"
Oct 31 01:58:51.946827 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.946789    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0b749c-921d-473d-98fd-c05c70279567" containerName="payload-processing"
Oct 31 01:58:51.953081 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.953054    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:51.959830 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.959767    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ac7e0d-openshift-default-dockercfg-frqwl\""
Oct 31 01:58:51.963540 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:51.963513    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"]
Oct 31 01:58:52.020896 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.020863    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-data\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.021133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.020930    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-token\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.021133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.020984    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-credential-socket\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.021133 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.021111    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-envoy\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.021345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.021206    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvsc\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-kube-api-access-wfvsc\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.021345 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.021322    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-socket\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.021455 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.021388    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-certs\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.021455 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.021420    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-podinfo\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.021573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.021555    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istiod-ca-cert\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.127968    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-token\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.128063    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-credential-socket\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.128151    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-envoy\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.128239    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvsc\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-kube-api-access-wfvsc\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.128345    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-socket\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.128406    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-certs\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.128435    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-podinfo\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.128545    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istiod-ca-cert\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.131065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.128602    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-data\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.155066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.132800    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-credential-socket\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.199115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.163862    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-envoy\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.199115 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.164900    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istiod-ca-cert\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.217091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.200333    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-data\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.217091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.200772    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-certs\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.217091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.200914    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-socket\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.217091 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.206592    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-token\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.253064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.249379    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-podinfo\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.279064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.274684    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvsc\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-kube-api-access-wfvsc\") pod \"e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") " pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.279415 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.279388    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:52.759971 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.759941    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"]
Oct 31 01:58:52.761211 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:58:52.761173    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc999384_bcd7_4dda_95ca_4ae3bf4fd130.slice/crio-dc34448bee719704c728a3c8f095241eaeb9820909aeb57038a210c3078bdf79 WatchSource:0}: Error finding container dc34448bee719704c728a3c8f095241eaeb9820909aeb57038a210c3078bdf79: Status 404 returned error can't find the container with id dc34448bee719704c728a3c8f095241eaeb9820909aeb57038a210c3078bdf79
Oct 31 01:58:52.764009 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.763976    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:52.764127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.764061    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:52.764127 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:52.764097    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:58:53.041727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:53.041640    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" event={"ID":"cc999384-bcd7-4dda-95ca-4ae3bf4fd130","Type":"ContainerStarted","Data":"1e8ad41d70f7333d215af8be4fff2feac8bc2168978c52af51c46036066df662"}
Oct 31 01:58:53.041727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:53.041701    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" event={"ID":"cc999384-bcd7-4dda-95ca-4ae3bf4fd130","Type":"ContainerStarted","Data":"dc34448bee719704c728a3c8f095241eaeb9820909aeb57038a210c3078bdf79"}
Oct 31 01:58:53.072436 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:53.072368    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podStartSLOduration=2.072349183 podStartE2EDuration="2.072349183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:53.064573297 +0000 UTC m=+2291.013221691" watchObservedRunningTime="2026-10-31 01:58:53.072349183 +0000 UTC m=+2291.020997580"
Oct 31 01:58:53.286068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:53.285808    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:53.367442 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:53.367361    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:54.068519 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:54.068485    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:58:54.069622 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:58:54.069579    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:59:05.623113 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:05.620093    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"]
Oct 31 01:59:05.625855 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:05.625835    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:05.632388 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:05.632361    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-89e955-dockercfg-9pm2k\""
Oct 31 01:59:05.632782 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:05.632762    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-89e955\""
Oct 31 01:59:05.641130 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:05.641106    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"]
Oct 31 01:59:05.761066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:05.748314    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfcdl\" (UniqueName: \"kubernetes.io/projected/240f0d9e-a0f1-4dd3-969e-287516d9aa97-kube-api-access-gfcdl\") pod \"payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn\" (UID: \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:05.787067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:05.761512    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/240f0d9e-a0f1-4dd3-969e-287516d9aa97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn\" (UID: \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:05.998365 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:05.964096    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfcdl\" (UniqueName: \"kubernetes.io/projected/240f0d9e-a0f1-4dd3-969e-287516d9aa97-kube-api-access-gfcdl\") pod \"payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn\" (UID: \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:06.016067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:06.002705    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/240f0d9e-a0f1-4dd3-969e-287516d9aa97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn\" (UID: \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:06.016382 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:06.004273    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/240f0d9e-a0f1-4dd3-969e-287516d9aa97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn\" (UID: \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:06.057714 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:06.057678    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfcdl\" (UniqueName: \"kubernetes.io/projected/240f0d9e-a0f1-4dd3-969e-287516d9aa97-kube-api-access-gfcdl\") pod \"payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn\" (UID: \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:06.248959 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:06.248871    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:06.663498 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:06.663406    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"]
Oct 31 01:59:07.423298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:07.422470    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn" event={"ID":"240f0d9e-a0f1-4dd3-969e-287516d9aa97","Type":"ContainerStarted","Data":"5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106"}
Oct 31 01:59:07.423298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:07.422525    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn" event={"ID":"240f0d9e-a0f1-4dd3-969e-287516d9aa97","Type":"ContainerStarted","Data":"4bbbb9ec6466aaf7384fc8dfdbc7f19c8c1fa6fb3f8e9e798a6b582488242957"}
Oct 31 01:59:07.423858 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:07.423592    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:10.427954 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:10.427885    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn" podStartSLOduration=5.427865165 podStartE2EDuration="5.427865165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:07.455509635 +0000 UTC m=+2305.404158027" watchObservedRunningTime="2026-10-31 01:59:10.427865165 +0000 UTC m=+2308.376513557"
Oct 31 01:59:10.429695 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:10.429672    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 01:59:12.054832 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:12.053252    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"]
Oct 31 01:59:12.062098 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:12.059387    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7143d18b-hb4hg"]
Oct 31 01:59:12.187542 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:59:12.187505    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:59:12.221066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:12.220721    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80979ac2-369d-45db-92fe-8039d0ed4787" path="/var/lib/kubelet/pods/80979ac2-369d-45db-92fe-8039d0ed4787/volumes"
Oct 31 01:59:19.507055 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:19.507005    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 01:59:21.030440 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:21.030404    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"]
Oct 31 01:59:21.035556 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:21.035525    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348ecd34-x26dq"]
Oct 31 01:59:22.175698 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.175660    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7dd4e85-4351-45c2-bc7f-84f0c383f916" path="/var/lib/kubelet/pods/b7dd4e85-4351-45c2-bc7f-84f0c383f916/volumes"
Oct 31 01:59:22.708507 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.708475    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"]
Oct 31 01:59:22.716663 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.716637    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.748294 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.745141    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0b6abf-openshift-default-dockercfg-xkp27\""
Oct 31 01:59:22.751924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.751897    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"]
Oct 31 01:59:22.827080 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.826999    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-kube-api-access-vg58c\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.827275 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.827258    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.827350 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.827309    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1321636-259d-41a9-9a08-99daca44e499-istio-podinfo\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.827516 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.827499    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-credential-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.827600 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.827584    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-istio-token\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.827716 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.827696    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-envoy\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.827802 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.827786    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-certs\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.827876 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.827861    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1321636-259d-41a9-9a08-99daca44e499-istiod-ca-cert\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.827930 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.827914    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-data\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930161    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1321636-259d-41a9-9a08-99daca44e499-istiod-ca-cert\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930234    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-data\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930285    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-kube-api-access-vg58c\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930418    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930445    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1321636-259d-41a9-9a08-99daca44e499-istio-podinfo\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930585    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-credential-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930664    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-istio-token\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930705    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-envoy\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.930786    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-certs\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.937247 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.931280    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-certs\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.971108 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.970782    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-data\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.971369 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.971341    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-credential-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.977856 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.972268    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.985062 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.981928    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1321636-259d-41a9-9a08-99daca44e499-istio-podinfo\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:22.985262 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:22.982688    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1321636-259d-41a9-9a08-99daca44e499-istiod-ca-cert\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:23.009648 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.009610    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-istio-token\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:23.025072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.019615    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-envoy\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:23.025072 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.020861    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-kube-api-access-vg58c\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:23.060375 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.060333    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:23.672473 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.672427    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"]
Oct 31 01:59:23.673204 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:59:23.673169    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1321636_259d_41a9_9a08_99daca44e499.slice/crio-60a4d00c110611040074e3375fe27a01a53c3c21c0c681eaa5ee12f23c3dccd2 WatchSource:0}: Error finding container 60a4d00c110611040074e3375fe27a01a53c3c21c0c681eaa5ee12f23c3dccd2: Status 404 returned error can't find the container with id 60a4d00c110611040074e3375fe27a01a53c3c21c0c681eaa5ee12f23c3dccd2
Oct 31 01:59:23.676824 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.676785    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:59:23.676924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.676871    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:59:23.676924 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.676916    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:59:23.970639 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.970553    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" event={"ID":"e1321636-259d-41a9-9a08-99daca44e499","Type":"ContainerStarted","Data":"a3a75510fa2a9792efd80a1b67515a5581db9c956c9d0130ff70cbda5e8afcfc"}
Oct 31 01:59:23.970866 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.970660    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" event={"ID":"e1321636-259d-41a9-9a08-99daca44e499","Type":"ContainerStarted","Data":"60a4d00c110611040074e3375fe27a01a53c3c21c0c681eaa5ee12f23c3dccd2"}
Oct 31 01:59:23.992993 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:23.992898    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podStartSLOduration=1.992877923 podStartE2EDuration="1.992877923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:23.992339743 +0000 UTC m=+2321.940988136" watchObservedRunningTime="2026-10-31 01:59:23.992877923 +0000 UTC m=+2321.941526316"
Oct 31 01:59:24.061781 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:24.061754    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:24.062965 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:24.062936    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 01:59:24.063142 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:24.062994    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 01:59:25.128573 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:25.128539    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:25.990438 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:25.990409    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:25.991934 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:25.991909    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 01:59:27.797657 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:27.797614    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"]
Oct 31 01:59:27.812317 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:27.812283    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:27.857359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:27.857319    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"]
Oct 31 01:59:27.885229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:27.862508    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0b6abf\""
Oct 31 01:59:27.885229 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:27.862614    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0b6abf-dockercfg-lhxmm\""
Oct 31 01:59:28.125894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:28.124102    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e2b864-2259-4446-89ad-0052cc9519b7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn\" (UID: \"43e2b864-2259-4446-89ad-0052cc9519b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:28.125894 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:28.124510    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx98x\" (UniqueName: \"kubernetes.io/projected/43e2b864-2259-4446-89ad-0052cc9519b7-kube-api-access-xx98x\") pod \"payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn\" (UID: \"43e2b864-2259-4446-89ad-0052cc9519b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:28.241313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:28.240862    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx98x\" (UniqueName: \"kubernetes.io/projected/43e2b864-2259-4446-89ad-0052cc9519b7-kube-api-access-xx98x\") pod \"payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn\" (UID: \"43e2b864-2259-4446-89ad-0052cc9519b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:28.241313 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:28.240955    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e2b864-2259-4446-89ad-0052cc9519b7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn\" (UID: \"43e2b864-2259-4446-89ad-0052cc9519b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:28.242102 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:28.242009    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e2b864-2259-4446-89ad-0052cc9519b7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn\" (UID: \"43e2b864-2259-4446-89ad-0052cc9519b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:28.257092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:28.257061    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx98x\" (UniqueName: \"kubernetes.io/projected/43e2b864-2259-4446-89ad-0052cc9519b7-kube-api-access-xx98x\") pod \"payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn\" (UID: \"43e2b864-2259-4446-89ad-0052cc9519b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:28.397443 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:28.395848    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:28.757403 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:28.757330    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"]
Oct 31 01:59:28.765438 ip-10-0-130-49 kubenswrapper[5293]: W1031 01:59:28.765411    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e2b864_2259_4446_89ad_0052cc9519b7.slice/crio-71b1b01dd144cee5b428209d472123cbe95485ae39f1431a17980c7a6348b5dc WatchSource:0}: Error finding container 71b1b01dd144cee5b428209d472123cbe95485ae39f1431a17980c7a6348b5dc: Status 404 returned error can't find the container with id 71b1b01dd144cee5b428209d472123cbe95485ae39f1431a17980c7a6348b5dc
Oct 31 01:59:29.310061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:29.308323    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn" event={"ID":"43e2b864-2259-4446-89ad-0052cc9519b7","Type":"ContainerStarted","Data":"92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc"}
Oct 31 01:59:29.310061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:29.308397    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn" event={"ID":"43e2b864-2259-4446-89ad-0052cc9519b7","Type":"ContainerStarted","Data":"71b1b01dd144cee5b428209d472123cbe95485ae39f1431a17980c7a6348b5dc"}
Oct 31 01:59:29.310061 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:29.309633    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:29.369065 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:29.363090    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn" podStartSLOduration=2.363033995 podStartE2EDuration="2.363033995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:29.343604204 +0000 UTC m=+2327.292252596" watchObservedRunningTime="2026-10-31 01:59:29.363033995 +0000 UTC m=+2327.311682388"
Oct 31 01:59:36.057770 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:36.057656    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"]
Oct 31 01:59:36.106067 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:36.095796    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-390dcf31-7dqbf"]
Oct 31 01:59:36.177298 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:36.177269    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67da2d4-da3b-4313-bd5e-cdc54c2778d8" path="/var/lib/kubelet/pods/b67da2d4-da3b-4313-bd5e-cdc54c2778d8/volumes"
Oct 31 01:59:41.331574 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:41.331546    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 01:59:41.779886 ip-10-0-130-49 kubenswrapper[5293]: E1031 01:59:41.779841    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 01:59:46.050727 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.050690    5293 scope.go:122] "RemoveContainer" containerID="9adf1cb7a4d1bb37bfee81c800712f9dd4b817a89494f28195151f17142e381c"
Oct 31 01:59:46.062030 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.062003    5293 scope.go:122] "RemoveContainer" containerID="feca8957f1d7d5e51f0d33a7b767326f96be8c7cefc761bbb140a688a1453801"
Oct 31 01:59:46.072854 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.072821    5293 scope.go:122] "RemoveContainer" containerID="244079c16ba8ee4bf147692c4fb9b9782996d88209a5450b5bc59e045b5d28ef"
Oct 31 01:59:46.083795 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.083768    5293 scope.go:122] "RemoveContainer" containerID="6ad7f49b6ece25cc19e5e7ec6e923d2f6c9a67c5d9c7063100a3df145b7da6a9"
Oct 31 01:59:46.093400 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.093373    5293 scope.go:122] "RemoveContainer" containerID="a68baec8a0345923487e92ffb9bba43145ea7a12c590d06627ef0ad8520a7638"
Oct 31 01:59:46.128770 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.128748    5293 scope.go:122] "RemoveContainer" containerID="067c6e241f9ccff770642c4763351c226da5bbeeab2c6e340d886034bc4b0400"
Oct 31 01:59:46.140092 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.140070    5293 scope.go:122] "RemoveContainer" containerID="906a26611aef4acf3d366fc1049e8e1ff88a1d0143e9bd1a94075f3edff8fdd3"
Oct 31 01:59:46.156521 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.156496    5293 scope.go:122] "RemoveContainer" containerID="1195d1dc5173a62c8a8ae615ffa962b40417c3184a7c3a7a35ca418300fee096"
Oct 31 01:59:46.168169 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.168147    5293 scope.go:122] "RemoveContainer" containerID="0ab0243639be49a493b84a0a34c4afd294281d22382e3b02853198fd60eed3a2"
Oct 31 01:59:46.178983 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:46.178938    5293 scope.go:122] "RemoveContainer" containerID="e2de11a6d059064c64a8bca80a0e24fbcf4194822ebe65438757ea236d371383"
Oct 31 01:59:47.681034 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.681005    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"]
Oct 31 01:59:47.687250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.687231    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.691359 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.691324    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:59:47.691509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.691422    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:59:47.691509 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.691432    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:59:47.691633 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.691603    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:59:47.706427 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.696509    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"]
Oct 31 01:59:47.841503 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.841472    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.841794 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.841769    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.841914 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.841853    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfjj\" (UniqueName: \"kubernetes.io/projected/61e2a839-b9a2-43c3-ad5b-b7aef672b314-kube-api-access-shfjj\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.948119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.946284    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.948119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.946379    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfjj\" (UniqueName: \"kubernetes.io/projected/61e2a839-b9a2-43c3-ad5b-b7aef672b314-kube-api-access-shfjj\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.948119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.946589    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.948119 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.947423    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.954271 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.954244    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:47.981063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:47.980277    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfjj\" (UniqueName: \"kubernetes.io/projected/61e2a839-b9a2-43c3-ad5b-b7aef672b314-kube-api-access-shfjj\") pod \"maas-api-revoke-sub-fad4bbf3-zl6zp\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:48.024064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:48.019338    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:48.049594 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:48.047121    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"]
Oct 31 01:59:48.055071 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:48.052075    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e0007c8-jmpvp"]
Oct 31 01:59:48.182506 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:48.182477    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d6020b-1edc-4c4b-b5e9-848635b65061" path="/var/lib/kubelet/pods/d9d6020b-1edc-4c4b-b5e9-848635b65061/volumes"
Oct 31 01:59:48.430621 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:48.430597    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"]
Oct 31 01:59:48.786494 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:48.786454    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp" event={"ID":"61e2a839-b9a2-43c3-ad5b-b7aef672b314","Type":"ContainerStarted","Data":"13760469f2272f5266fe44adff8dbc192862b12f96b81bd5b413f032494f3bb8"}
Oct 31 01:59:49.824258 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:49.824223    5293 generic.go:356] "Generic (PLEG): container finished" podID="61e2a839-b9a2-43c3-ad5b-b7aef672b314" containerID="8cb904b5a68dbd206ee7d28f1cc6fe6c935c532c93d34a004354fed3498051f3" exitCode=0
Oct 31 01:59:49.825398 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:49.825368    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp" event={"ID":"61e2a839-b9a2-43c3-ad5b-b7aef672b314","Type":"ContainerDied","Data":"8cb904b5a68dbd206ee7d28f1cc6fe6c935c532c93d34a004354fed3498051f3"}
Oct 31 01:59:50.038219 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:50.038131    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"]
Oct 31 01:59:50.077066 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:50.071718    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637cedcd-k798b"]
Oct 31 01:59:50.234238 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:50.234200    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5bf6ea-fce6-47bb-a77f-37bd64ac7453" path="/var/lib/kubelet/pods/dd5bf6ea-fce6-47bb-a77f-37bd64ac7453/volumes"
Oct 31 01:59:51.214193 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.214167    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:51.329218 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.329162    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-openshift-ca\") pod \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") "
Oct 31 01:59:51.329586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.329249    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-xks-ca\") pod \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") "
Oct 31 01:59:51.329586 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.329346    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e2a839-b9a2-43c3-ad5b-b7aef672b314-kube-api-access-shfjj\" (UniqueName: \"kubernetes.io/projected/61e2a839-b9a2-43c3-ad5b-b7aef672b314-kube-api-access-shfjj\") pod \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\" (UID: \"61e2a839-b9a2-43c3-ad5b-b7aef672b314\") "
Oct 31 01:59:51.330131 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.329829    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-openshift-ca" pod "61e2a839-b9a2-43c3-ad5b-b7aef672b314" (UID: "61e2a839-b9a2-43c3-ad5b-b7aef672b314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:59:51.330437 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.330303    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:59:51.333195 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.333168    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-xks-ca" pod "61e2a839-b9a2-43c3-ad5b-b7aef672b314" (UID: "61e2a839-b9a2-43c3-ad5b-b7aef672b314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:59:51.333779 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.333753    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e2a839-b9a2-43c3-ad5b-b7aef672b314-kube-api-access-shfjj" pod "61e2a839-b9a2-43c3-ad5b-b7aef672b314" (UID: "61e2a839-b9a2-43c3-ad5b-b7aef672b314"). InnerVolumeSpecName "kube-api-access-shfjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:51.432828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.432738    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e2a839-b9a2-43c3-ad5b-b7aef672b314-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:59:51.432828 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.432779    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shfjj\" (UniqueName: \"kubernetes.io/projected/61e2a839-b9a2-43c3-ad5b-b7aef672b314-kube-api-access-shfjj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:59:51.849576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.849501    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp" event={"ID":"61e2a839-b9a2-43c3-ad5b-b7aef672b314","Type":"ContainerDied","Data":"13760469f2272f5266fe44adff8dbc192862b12f96b81bd5b413f032494f3bb8"}
Oct 31 01:59:51.849576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.849549    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13760469f2272f5266fe44adff8dbc192862b12f96b81bd5b413f032494f3bb8"
Oct 31 01:59:51.849576 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:51.849548    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"
Oct 31 01:59:52.026853 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:52.026765    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"]
Oct 31 01:59:52.030628 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:52.030598    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09606703-gwxls"]
Oct 31 01:59:52.196175 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:52.194310    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef4f950-cf22-4f06-a514-d051a1634334" path="/var/lib/kubelet/pods/5ef4f950-cf22-4f06-a514-d051a1634334/volumes"
Oct 31 01:59:55.109197 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:55.109157    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:59:55.109753 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:55.109270    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:59:55.962381 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:55.962303    5293 generic.go:356] "Generic (PLEG): container finished" podID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerID="1e8ad41d70f7333d215af8be4fff2feac8bc2168978c52af51c46036066df662" exitCode=0
Oct 31 01:59:55.962627 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:55.962488    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" event={"ID":"cc999384-bcd7-4dda-95ca-4ae3bf4fd130","Type":"ContainerDied","Data":"1e8ad41d70f7333d215af8be4fff2feac8bc2168978c52af51c46036066df662"}
Oct 31 01:59:55.970285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:55.962979    5293 scope.go:122] "RemoveContainer" containerID="1e8ad41d70f7333d215af8be4fff2feac8bc2168978c52af51c46036066df662"
Oct 31 01:59:55.970285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:55.965290    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:59:55.970285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:55.965376    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:59:55.970285 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:55.965423    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:59:57.087064 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.086478    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" event={"ID":"cc999384-bcd7-4dda-95ca-4ae3bf4fd130","Type":"ContainerStarted","Data":"93ea254fdbb5a5d032e80d154e46a029f66eb6ee7d8ecc3503908046fa4c7c42"}
Oct 31 01:59:57.094063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.091896    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"]
Oct 31 01:59:57.111068 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.107464    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16611557-zkw2q"]
Oct 31 01:59:57.280413 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.280383    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 01:59:57.280705 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.280686    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:59:57.280783 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.280727    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 01:59:57.934988 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.934948    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"]
Oct 31 01:59:57.935699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.935680    5293 cpu_manager.go:414] "containerMap: removing container" podUID="61e2a839-b9a2-43c3-ad5b-b7aef672b314" containerName="revoke-keys"
Oct 31 01:59:57.935699 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.935698    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e2a839-b9a2-43c3-ad5b-b7aef672b314" containerName="revoke-keys"
Oct 31 01:59:57.935856 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.935812    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e2a839-b9a2-43c3-ad5b-b7aef672b314" containerName="revoke-keys"
Oct 31 01:59:57.940290 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.940270    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:57.947173 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.947149    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:59:57.950929 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.947771    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:59:57.950929 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.950213    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"]
Oct 31 01:59:57.950929 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.950291    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:59:57.981250 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:57.953466    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:59:58.243786 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.243682    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e166a3a5-9fc7-448f-b0a7-f5792ae2d14e" path="/var/lib/kubelet/pods/e166a3a5-9fc7-448f-b0a7-f5792ae2d14e/volumes"
Oct 31 01:59:58.254728 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.254693    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.254893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.254815    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxcn\" (UniqueName: \"kubernetes.io/projected/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-kube-api-access-gwxcn\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.254893 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.254836    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.280753 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.280719    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:59:58.280899 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.280783    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 01:59:58.356023 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.355982    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.356283 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.356179    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxcn\" (UniqueName: \"kubernetes.io/projected/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-kube-api-access-gwxcn\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.356283 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.356214    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.356985 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.356910    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.358873 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.358849    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.369637 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.369614    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxcn\" (UniqueName: \"kubernetes.io/projected/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-kube-api-access-gwxcn\") pod \"maas-api-revoke-sub-7deaa4bc-f5bbf\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:58.479342 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:58.479307    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 01:59:59.223063 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:59.222845    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"]
Oct 31 01:59:59.296865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:59.296389    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:59:59.296865 ip-10-0-130-49 kubenswrapper[5293]: I1031 01:59:59.296452    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 02:00:00.282847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.282455    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 02:00:00.286392 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.283094    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 02:00:00.485954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.485924    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"]
Oct 31 02:00:00.486482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.485968    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42475bd4-trm9g"]
Oct 31 02:00:00.486482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.486117    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"]
Oct 31 02:00:00.491273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.491168    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf" event={"ID":"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852","Type":"ContainerStarted","Data":"05609646e758487db92930ac86d50aa69411d81a4285704e914af1eb03ee4735"}
Oct 31 02:00:00.491542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.491521    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf" event={"ID":"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852","Type":"ContainerStarted","Data":"661b2754c05a13db5a2205525a9d98fbc8270acf9304fc1255d33691f143ed9b"}
Oct 31 02:00:00.491732 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.491715    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"]
Oct 31 02:00:00.491838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.491815    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:00.539534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.539500    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"]
Oct 31 02:00:00.539878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.539855    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:00.546313 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.546286    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl\" (UID: \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:00.546441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.546365    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnz4t\" (UniqueName: \"kubernetes.io/projected/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-kube-api-access-fnz4t\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl\" (UID: \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:00.549743 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.547220    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"]
Oct 31 02:00:00.549743 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.549204    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:00.556618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.556602    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"]
Oct 31 02:00:00.556767 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.556757    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"]
Oct 31 02:00:00.557451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.557430    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:00.563143 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.563127    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"]
Oct 31 02:00:00.563379 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.563367    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"]
Oct 31 02:00:00.563506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.563496    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"]
Oct 31 02:00:00.563651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.563330    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:00.568220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.568203    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"]
Oct 31 02:00:00.568842 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.568828    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:00.573693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.573674    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"]
Oct 31 02:00:00.573834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.573814    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:00.578115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.578101    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"]
Oct 31 02:00:00.578231 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.578212    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:00.586222 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.586202    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"]
Oct 31 02:00:00.586437 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.586419    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:00.591485 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.591466    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"]
Oct 31 02:00:00.591593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.591499    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"]
Oct 31 02:00:00.591593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.591515    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"]
Oct 31 02:00:00.591593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.591572    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:00.592670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.592565    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"]
Oct 31 02:00:00.595713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.595686    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"]
Oct 31 02:00:00.640925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.640897    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"]
Oct 31 02:00:00.647355 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647335    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"]
Oct 31 02:00:00.647532 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647475    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a726d42d-1730-47aa-8a2d-1b453e4f4bab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8\" (UID: \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:00.647644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647595    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtd44\" (UniqueName: \"kubernetes.io/projected/35bf319f-dfea-458f-be77-6c0823836fb5-kube-api-access-qtd44\") pod \"maas-api-key-cleanup-29890200-5ptbb\" (UID: \"35bf319f-dfea-458f-be77-6c0823836fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:00.647686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647642    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pldgj\" (UniqueName: \"kubernetes.io/projected/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-kube-api-access-pldgj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt\" (UID: \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:00.647686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647662    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bf319f-dfea-458f-be77-6c0823836fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-5ptbb\" (UID: \"35bf319f-dfea-458f-be77-6c0823836fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:00.647686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647683    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5627e74-5b80-493a-bc75-2e42e13942cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9\" (UID: \"e5627e74-5b80-493a-bc75-2e42e13942cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:00.647795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647703    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c48l\" (UniqueName: \"kubernetes.io/projected/a726d42d-1730-47aa-8a2d-1b453e4f4bab-kube-api-access-4c48l\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8\" (UID: \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:00.647795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647724    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3681d17a-a7c0-45ac-9888-d44c117fc742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd\" (UID: \"3681d17a-a7c0-45ac-9888-d44c117fc742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:00.647795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647750    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt\" (UID: \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:00.647888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647797    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252vv\" (UniqueName: \"kubernetes.io/projected/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-kube-api-access-252vv\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd\" (UID: \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:00.647888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647815    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/227a89a1-09f8-4819-8526-1340b1e8ca1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l\" (UID: \"227a89a1-09f8-4819-8526-1340b1e8ca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:00.647888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647846    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk\" (UID: \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:00.647888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647873    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6v5s\" (UniqueName: \"kubernetes.io/projected/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-kube-api-access-b6v5s\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk\" (UID: \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:00.648022 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647893    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcx7r\" (UniqueName: \"kubernetes.io/projected/3681d17a-a7c0-45ac-9888-d44c117fc742-kube-api-access-vcx7r\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd\" (UID: \"3681d17a-a7c0-45ac-9888-d44c117fc742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:00.648022 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647927    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-784qq\" (UniqueName: \"kubernetes.io/projected/e5627e74-5b80-493a-bc75-2e42e13942cb-kube-api-access-784qq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9\" (UID: \"e5627e74-5b80-493a-bc75-2e42e13942cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:00.648022 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647949    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl\" (UID: \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:00.648022 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.647989    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnz4t\" (UniqueName: \"kubernetes.io/projected/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-kube-api-access-fnz4t\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl\" (UID: \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:00.648177 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.648034    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7dv\" (UniqueName: \"kubernetes.io/projected/aecd11ba-cc8a-48bb-8641-a0089eadbc07-kube-api-access-zc7dv\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm\" (UID: \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:00.648177 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.648102    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd\" (UID: \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:00.648177 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.648132    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85vw\" (UniqueName: \"kubernetes.io/projected/227a89a1-09f8-4819-8526-1340b1e8ca1d-kube-api-access-g85vw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l\" (UID: \"227a89a1-09f8-4819-8526-1340b1e8ca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:00.648177 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.648165    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecd11ba-cc8a-48bb-8641-a0089eadbc07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm\" (UID: \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:00.648795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.648772    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl\" (UID: \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:00.648869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.648772    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf" podStartSLOduration=3.648754908 podStartE2EDuration="3.648754908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:00.521889128 +0000 UTC m=+2358.470537525" watchObservedRunningTime="2026-10-31 02:00:00.648754908 +0000 UTC m=+2358.597403302"
Oct 31 02:00:00.666526 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.666474    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnz4t\" (UniqueName: \"kubernetes.io/projected/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-kube-api-access-fnz4t\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl\" (UID: \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:00.749961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.749892    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3681d17a-a7c0-45ac-9888-d44c117fc742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd\" (UID: \"3681d17a-a7c0-45ac-9888-d44c117fc742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:00.750227 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.750205    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt\" (UID: \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:00.750399 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.750385    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252vv\" (UniqueName: \"kubernetes.io/projected/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-kube-api-access-252vv\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd\" (UID: \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:00.750508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.750494    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/227a89a1-09f8-4819-8526-1340b1e8ca1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l\" (UID: \"227a89a1-09f8-4819-8526-1340b1e8ca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:00.750621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.750608    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk\" (UID: \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:00.750735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.750721    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6v5s\" (UniqueName: \"kubernetes.io/projected/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-kube-api-access-b6v5s\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk\" (UID: \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:00.750828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.750816    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcx7r\" (UniqueName: \"kubernetes.io/projected/3681d17a-a7c0-45ac-9888-d44c117fc742-kube-api-access-vcx7r\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd\" (UID: \"3681d17a-a7c0-45ac-9888-d44c117fc742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:00.750950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.750936    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784qq\" (UniqueName: \"kubernetes.io/projected/e5627e74-5b80-493a-bc75-2e42e13942cb-kube-api-access-784qq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9\" (UID: \"e5627e74-5b80-493a-bc75-2e42e13942cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:00.751140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751125    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7dv\" (UniqueName: \"kubernetes.io/projected/aecd11ba-cc8a-48bb-8641-a0089eadbc07-kube-api-access-zc7dv\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm\" (UID: \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:00.751239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751227    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd\" (UID: \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:00.751357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751345    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85vw\" (UniqueName: \"kubernetes.io/projected/227a89a1-09f8-4819-8526-1340b1e8ca1d-kube-api-access-g85vw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l\" (UID: \"227a89a1-09f8-4819-8526-1340b1e8ca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:00.751471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751458    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecd11ba-cc8a-48bb-8641-a0089eadbc07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm\" (UID: \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:00.751621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751593    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/227a89a1-09f8-4819-8526-1340b1e8ca1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l\" (UID: \"227a89a1-09f8-4819-8526-1340b1e8ca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:00.751671 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751577    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt\" (UID: \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:00.751721 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751545    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3681d17a-a7c0-45ac-9888-d44c117fc742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd\" (UID: \"3681d17a-a7c0-45ac-9888-d44c117fc742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:00.751832 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751818    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a726d42d-1730-47aa-8a2d-1b453e4f4bab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8\" (UID: \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:00.751956 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.751947    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtd44\" (UniqueName: \"kubernetes.io/projected/35bf319f-dfea-458f-be77-6c0823836fb5-kube-api-access-qtd44\") pod \"maas-api-key-cleanup-29890200-5ptbb\" (UID: \"35bf319f-dfea-458f-be77-6c0823836fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:00.752091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.752080    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pldgj\" (UniqueName: \"kubernetes.io/projected/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-kube-api-access-pldgj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt\" (UID: \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:00.752174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.752165    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bf319f-dfea-458f-be77-6c0823836fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-5ptbb\" (UID: \"35bf319f-dfea-458f-be77-6c0823836fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:00.752243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.752234    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5627e74-5b80-493a-bc75-2e42e13942cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9\" (UID: \"e5627e74-5b80-493a-bc75-2e42e13942cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:00.752320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.752305    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c48l\" (UniqueName: \"kubernetes.io/projected/a726d42d-1730-47aa-8a2d-1b453e4f4bab-kube-api-access-4c48l\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8\" (UID: \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:00.752584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.752562    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd\" (UID: \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:00.752719 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.752702    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk\" (UID: \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:00.753243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.752247    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecd11ba-cc8a-48bb-8641-a0089eadbc07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm\" (UID: \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:00.753482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.753461    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bf319f-dfea-458f-be77-6c0823836fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-5ptbb\" (UID: \"35bf319f-dfea-458f-be77-6c0823836fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:00.753954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.753789    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5627e74-5b80-493a-bc75-2e42e13942cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9\" (UID: \"e5627e74-5b80-493a-bc75-2e42e13942cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:00.754070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.753973    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a726d42d-1730-47aa-8a2d-1b453e4f4bab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8\" (UID: \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:00.766009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.765980    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252vv\" (UniqueName: \"kubernetes.io/projected/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-kube-api-access-252vv\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd\" (UID: \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:00.766560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.766476    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6v5s\" (UniqueName: \"kubernetes.io/projected/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-kube-api-access-b6v5s\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk\" (UID: \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:00.766560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.766535    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcx7r\" (UniqueName: \"kubernetes.io/projected/3681d17a-a7c0-45ac-9888-d44c117fc742-kube-api-access-vcx7r\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd\" (UID: \"3681d17a-a7c0-45ac-9888-d44c117fc742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:00.766682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.766661    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtd44\" (UniqueName: \"kubernetes.io/projected/35bf319f-dfea-458f-be77-6c0823836fb5-kube-api-access-qtd44\") pod \"maas-api-key-cleanup-29890200-5ptbb\" (UID: \"35bf319f-dfea-458f-be77-6c0823836fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:00.768841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.768818    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-784qq\" (UniqueName: \"kubernetes.io/projected/e5627e74-5b80-493a-bc75-2e42e13942cb-kube-api-access-784qq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9\" (UID: \"e5627e74-5b80-493a-bc75-2e42e13942cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:00.769075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.769055    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c48l\" (UniqueName: \"kubernetes.io/projected/a726d42d-1730-47aa-8a2d-1b453e4f4bab-kube-api-access-4c48l\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8\" (UID: \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:00.769234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.769218    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85vw\" (UniqueName: \"kubernetes.io/projected/227a89a1-09f8-4819-8526-1340b1e8ca1d-kube-api-access-g85vw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l\" (UID: \"227a89a1-09f8-4819-8526-1340b1e8ca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:00.769403 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.769382    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pldgj\" (UniqueName: \"kubernetes.io/projected/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-kube-api-access-pldgj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt\" (UID: \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:00.771487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.771465    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7dv\" (UniqueName: \"kubernetes.io/projected/aecd11ba-cc8a-48bb-8641-a0089eadbc07-kube-api-access-zc7dv\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm\" (UID: \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:00.868096 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.859161    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:00.899067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.873276    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:00.899067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.895814    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:00.899067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:00.896071    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:01.072112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.042795    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:01.072112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.057608    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:01.184481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.184441    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:01.347078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.342678    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 02:00:01.514077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.506589    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:01.514550 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.514171    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 02:00:01.748213 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.748182    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"]
Oct 31 02:00:01.880445 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.860640    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:00:01.880445 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.878664    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:01.886480 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.885974    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:01.948272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.947965    5293 generic.go:356] "Generic (PLEG): container finished" podID="1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852" containerID="05609646e758487db92930ac86d50aa69411d81a4285704e914af1eb03ee4735" exitCode=0
Oct 31 02:00:01.951854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.951821    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852" containerName="revoke-keys"
Oct 31 02:00:01.952061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.952028    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852" containerName="revoke-keys"
Oct 31 02:00:01.963234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.963204    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"]
Oct 31 02:00:01.963570 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.963543    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf" event={"ID":"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852","Type":"ContainerDied","Data":"05609646e758487db92930ac86d50aa69411d81a4285704e914af1eb03ee4735"}
Oct 31 02:00:01.964161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.964140    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:01.977605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.976219    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-e2e--c450cd8d-dockercfg-2j4lf\""
Oct 31 02:00:01.977605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.976560    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-09706649fc65\""
Oct 31 02:00:01.977605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.976632    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0b6abf\""
Oct 31 02:00:01.977605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.976738    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"routing-overlay\""
Oct 31 02:00:01.977605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.976799    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"kube-root-ca.crt\""
Oct 31 02:00:01.977605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:01.976876    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"openshift-service-ca.crt\""
Oct 31 02:00:02.087693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.087370    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbbl\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-kube-api-access-xrbbl\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.087693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.087641    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.087829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.087766    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.088161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.087899    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.190925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.190030    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrbbl\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-kube-api-access-xrbbl\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.190925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.190206    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.190925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.190278    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.190925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.190378    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.192092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.192029    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.192727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.192681    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.246119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.234405    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1791185-a8ab-4338-ba0f-a5d40aff5208" path="/var/lib/kubelet/pods/c1791185-a8ab-4338-ba0f-a5d40aff5208/volumes"
Oct 31 02:00:02.257269 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.257233    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.261180 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.260927    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrbbl\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-kube-api-access-xrbbl\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.284915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.281066    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 02:00:02.284915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.281118    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 02:00:02.507768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.507736    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:02.847221 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.847148    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"]
Oct 31 02:00:02.854551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.854347    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"]
Oct 31 02:00:02.862922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.862897    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"]
Oct 31 02:00:02.945834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.944829    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"]
Oct 31 02:00:02.950737 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.950575    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"]
Oct 31 02:00:02.962153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.959863    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" event={"ID":"3681d17a-a7c0-45ac-9888-d44c117fc742","Type":"ContainerStarted","Data":"60bb6eacd8a333d8633c1b65d6407e09606a6c6edd55b5e471d761efee47ee7b"}
Oct 31 02:00:02.962153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.961982    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" event={"ID":"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6","Type":"ContainerStarted","Data":"b292f290e44a49df38b1bd43b4842453e61651f453c924cb888f990e238b0617"}
Oct 31 02:00:02.965800 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.965776    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" event={"ID":"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660","Type":"ContainerStarted","Data":"adc08c2dbf65582cd1424ed065f473b048d4e58260f4a2847cb75ba3cafe8073"}
Oct 31 02:00:02.968393 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:00:02.968358    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecd11ba_cc8a_48bb_8641_a0089eadbc07.slice/crio-fcf45653bc58ea8c1ba7b4686089d916f32dac12ab1fb13dfcb54d9253e6448a WatchSource:0}: Error finding container fcf45653bc58ea8c1ba7b4686089d916f32dac12ab1fb13dfcb54d9253e6448a: Status 404 returned error can't find the container with id fcf45653bc58ea8c1ba7b4686089d916f32dac12ab1fb13dfcb54d9253e6448a
Oct 31 02:00:02.996184 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:02.996154    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"]
Oct 31 02:00:03.274595 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.274571    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"]
Oct 31 02:00:03.282372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.282319    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 02:00:03.282461 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.282365    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 02:00:03.358958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.358925    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"]
Oct 31 02:00:03.394992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.394965    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"]
Oct 31 02:00:03.460584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.460118    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"]
Oct 31 02:00:03.475400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.475085    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"]
Oct 31 02:00:03.750580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.750551    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 02:00:03.774344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.774318    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-xks-ca\") pod \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") "
Oct 31 02:00:03.774472 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.774401    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-kube-api-access-gwxcn\" (UniqueName: \"kubernetes.io/projected/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-kube-api-access-gwxcn\") pod \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") "
Oct 31 02:00:03.774596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.774579    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-openshift-ca\") pod \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\" (UID: \"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852\") "
Oct 31 02:00:03.775020 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.774991    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-openshift-ca" pod "1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852" (UID: "1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:03.783603 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.783567    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-xks-ca" pod "1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852" (UID: "1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:00:03.785602 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.785572    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-kube-api-access-gwxcn" pod "1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852" (UID: "1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852"). InnerVolumeSpecName "kube-api-access-gwxcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:03.891065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.877617    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxcn\" (UniqueName: \"kubernetes.io/projected/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-kube-api-access-gwxcn\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:03.891065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.877660    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:03.891065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:03.877676    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:04.038455 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.034992    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf" event={"ID":"1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852","Type":"ContainerDied","Data":"661b2754c05a13db5a2205525a9d98fbc8270acf9304fc1255d33691f143ed9b"}
Oct 31 02:00:04.038455 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.035064    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="661b2754c05a13db5a2205525a9d98fbc8270acf9304fc1255d33691f143ed9b"
Oct 31 02:00:04.038455 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.035130    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"
Oct 31 02:00:04.046239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.045426    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" event={"ID":"3681d17a-a7c0-45ac-9888-d44c117fc742","Type":"ContainerStarted","Data":"b88630a9e94b2482f8954e94213162a006301825fe85330998bb023972e26648"}
Oct 31 02:00:04.053261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.053214    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" event={"ID":"227a89a1-09f8-4819-8526-1340b1e8ca1d","Type":"ContainerStarted","Data":"526c33758ae88d82cbe4e806ff5641bb4b6acaf97efab7db064807079778df29"}
Oct 31 02:00:04.055591 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.054892    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" event={"ID":"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6","Type":"ContainerStarted","Data":"db8c596df3465482aa31ce165028a0fcc892472a46c36547f239082fbc706863"}
Oct 31 02:00:04.057267 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.057233    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" event={"ID":"35bf319f-dfea-458f-be77-6c0823836fb5","Type":"ContainerStarted","Data":"84107401bae0abef5add45c4add43cdd03816518e70fec5dde2ac36868bc696d"}
Oct 31 02:00:04.059712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.059650    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" event={"ID":"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e","Type":"ContainerStarted","Data":"b6cec7bf911f1ba2fe570fc1f2e9cfc1c43f49f91c655ee7f592edf022d98f28"}
Oct 31 02:00:04.063078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.062264    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" event={"ID":"e5627e74-5b80-493a-bc75-2e42e13942cb","Type":"ContainerStarted","Data":"313989a89c05d9492de7ed7b61425a7eda70ccdd5c8fd19f4b52cc5a6ca6bc96"}
Oct 31 02:00:04.066078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.063920    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" event={"ID":"a726d42d-1730-47aa-8a2d-1b453e4f4bab","Type":"ContainerStarted","Data":"1c7de1016673b63b7d7b31d7c8e62a418c74a7eb438fa7f0f7a7d5b69d68eedd"}
Oct 31 02:00:04.079339 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.078160    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht" event={"ID":"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d","Type":"ContainerStarted","Data":"ea3de0d1dc9601331ad27684318c25989682daaaadf1753b40e6ca9aab3ffd33"}
Oct 31 02:00:04.117345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.115379    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" podStartSLOduration=4.115362386 podStartE2EDuration="4.115362386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:04.064258113 +0000 UTC m=+2362.012906506" watchObservedRunningTime="2026-10-31 02:00:04.115362386 +0000 UTC m=+2362.064010780"
Oct 31 02:00:04.125552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.125498    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" event={"ID":"aecd11ba-cc8a-48bb-8641-a0089eadbc07","Type":"ContainerStarted","Data":"fcf45653bc58ea8c1ba7b4686089d916f32dac12ab1fb13dfcb54d9253e6448a"}
Oct 31 02:00:04.128728 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.128706    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" event={"ID":"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a","Type":"ContainerStarted","Data":"168439cce9149334c545b928c1d2530f2b94ae8e6acdab6f776b9eea336b4e70"}
Oct 31 02:00:04.137961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.134211    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" podStartSLOduration=4.134195056 podStartE2EDuration="4.134195056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:04.115266849 +0000 UTC m=+2362.063915243" watchObservedRunningTime="2026-10-31 02:00:04.134195056 +0000 UTC m=+2362.082843446"
Oct 31 02:00:04.137961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.136804    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" event={"ID":"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660","Type":"ContainerStarted","Data":"8abdbaa4a5c6805c300a61e35d70a49ba881f0af7cf66d3c93c975055f94464e"}
Oct 31 02:00:04.138682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.138646    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" podStartSLOduration=4.138634201 podStartE2EDuration="4.138634201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:04.130763089 +0000 UTC m=+2362.079411482" watchObservedRunningTime="2026-10-31 02:00:04.138634201 +0000 UTC m=+2362.087282603"
Oct 31 02:00:04.156938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.156899    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" podStartSLOduration=4.156887258 podStartE2EDuration="4.156887258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:04.152105073 +0000 UTC m=+2362.100753467" watchObservedRunningTime="2026-10-31 02:00:04.156887258 +0000 UTC m=+2362.105535649"
Oct 31 02:00:04.167901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.167863    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" podStartSLOduration=4.167849871 podStartE2EDuration="4.167849871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:04.167825647 +0000 UTC m=+2362.116474040" watchObservedRunningTime="2026-10-31 02:00:04.167849871 +0000 UTC m=+2362.116498263"
Oct 31 02:00:04.286059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.285969    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 02:00:04.286497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.286476    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 02:00:04.287209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:04.287193    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 02:00:05.148438 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.148404    5293 generic.go:356] "Generic (PLEG): container finished" podID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerID="34f9f11148c88e82bf21ec2afa6911ae731e8ba48b490d01d5090b38ae65d7f5" exitCode=60
Oct 31 02:00:05.148825 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.148488    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" event={"ID":"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e","Type":"ContainerDied","Data":"34f9f11148c88e82bf21ec2afa6911ae731e8ba48b490d01d5090b38ae65d7f5"}
Oct 31 02:00:05.148871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.148827    5293 scope.go:122] "RemoveContainer" containerID="34f9f11148c88e82bf21ec2afa6911ae731e8ba48b490d01d5090b38ae65d7f5"
Oct 31 02:00:05.150778 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.150714    5293 generic.go:356] "Generic (PLEG): container finished" podID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerID="c712ba8a99459d8d0daff8e7c5bc583c1b2a82f1ca6b431fb0252b4cfb249957" exitCode=60
Oct 31 02:00:05.150864 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.150781    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" event={"ID":"e5627e74-5b80-493a-bc75-2e42e13942cb","Type":"ContainerDied","Data":"c712ba8a99459d8d0daff8e7c5bc583c1b2a82f1ca6b431fb0252b4cfb249957"}
Oct 31 02:00:05.151119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.151098    5293 scope.go:122] "RemoveContainer" containerID="c712ba8a99459d8d0daff8e7c5bc583c1b2a82f1ca6b431fb0252b4cfb249957"
Oct 31 02:00:05.155451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.155433    5293 generic.go:356] "Generic (PLEG): container finished" podID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerID="af422274289d0321d814427bb13a4b2fffddadc6ef7ee6598d5bebe69e227ed9" exitCode=60
Oct 31 02:00:05.155539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.155458    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" event={"ID":"a726d42d-1730-47aa-8a2d-1b453e4f4bab","Type":"ContainerDied","Data":"af422274289d0321d814427bb13a4b2fffddadc6ef7ee6598d5bebe69e227ed9"}
Oct 31 02:00:05.155776 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.155759    5293 scope.go:122] "RemoveContainer" containerID="af422274289d0321d814427bb13a4b2fffddadc6ef7ee6598d5bebe69e227ed9"
Oct 31 02:00:05.157236 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.157212    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht" event={"ID":"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d","Type":"ContainerStarted","Data":"6110044fb2ef081435b22c24e4bd4bb259de99e72fa912123f93f6e99a59a64c"}
Oct 31 02:00:05.157349 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.157266    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:05.158887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.158868    5293 generic.go:356] "Generic (PLEG): container finished" podID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerID="897eabf033e1548ba38c0acd079dac9d50536e3e3872447c8da7d5984ed9b1be" exitCode=60
Oct 31 02:00:05.158980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.158956    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" event={"ID":"aecd11ba-cc8a-48bb-8641-a0089eadbc07","Type":"ContainerDied","Data":"897eabf033e1548ba38c0acd079dac9d50536e3e3872447c8da7d5984ed9b1be"}
Oct 31 02:00:05.159212 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.159195    5293 scope.go:122] "RemoveContainer" containerID="897eabf033e1548ba38c0acd079dac9d50536e3e3872447c8da7d5984ed9b1be"
Oct 31 02:00:05.161583 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.161541    5293 generic.go:356] "Generic (PLEG): container finished" podID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerID="faa0cc1be8590f8bc68564da9b4abbd4f0e2c7db6b226021875a67dc74afcabe" exitCode=60
Oct 31 02:00:05.161676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.161590    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" event={"ID":"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a","Type":"ContainerDied","Data":"faa0cc1be8590f8bc68564da9b4abbd4f0e2c7db6b226021875a67dc74afcabe"}
Oct 31 02:00:05.162078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.161815    5293 scope.go:122] "RemoveContainer" containerID="faa0cc1be8590f8bc68564da9b4abbd4f0e2c7db6b226021875a67dc74afcabe"
Oct 31 02:00:05.164457 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.163830    5293 generic.go:356] "Generic (PLEG): container finished" podID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerID="8abdbaa4a5c6805c300a61e35d70a49ba881f0af7cf66d3c93c975055f94464e" exitCode=60
Oct 31 02:00:05.164457 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.163911    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" event={"ID":"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660","Type":"ContainerDied","Data":"8abdbaa4a5c6805c300a61e35d70a49ba881f0af7cf66d3c93c975055f94464e"}
Oct 31 02:00:05.164457 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.164144    5293 scope.go:122] "RemoveContainer" containerID="8abdbaa4a5c6805c300a61e35d70a49ba881f0af7cf66d3c93c975055f94464e"
Oct 31 02:00:05.169263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.169243    5293 generic.go:356] "Generic (PLEG): container finished" podID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerID="b88630a9e94b2482f8954e94213162a006301825fe85330998bb023972e26648" exitCode=60
Oct 31 02:00:05.169386 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.169367    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" event={"ID":"3681d17a-a7c0-45ac-9888-d44c117fc742","Type":"ContainerDied","Data":"b88630a9e94b2482f8954e94213162a006301825fe85330998bb023972e26648"}
Oct 31 02:00:05.169526 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.169509    5293 scope.go:122] "RemoveContainer" containerID="b88630a9e94b2482f8954e94213162a006301825fe85330998bb023972e26648"
Oct 31 02:00:05.177937 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.177907    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" event={"ID":"227a89a1-09f8-4819-8526-1340b1e8ca1d","Type":"ContainerStarted","Data":"4768d0aaeab126e5713786ea2f22d489cb89636dbb49914eff59873d98dd7125"}
Oct 31 02:00:05.180193 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.180169    5293 generic.go:356] "Generic (PLEG): container finished" podID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerID="db8c596df3465482aa31ce165028a0fcc892472a46c36547f239082fbc706863" exitCode=60
Oct 31 02:00:05.180298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.180237    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" event={"ID":"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6","Type":"ContainerDied","Data":"db8c596df3465482aa31ce165028a0fcc892472a46c36547f239082fbc706863"}
Oct 31 02:00:05.183953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.183933    5293 scope.go:122] "RemoveContainer" containerID="db8c596df3465482aa31ce165028a0fcc892472a46c36547f239082fbc706863"
Oct 31 02:00:05.187134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.186347    5293 generic.go:356] "Generic (PLEG): container finished" podID="35bf319f-dfea-458f-be77-6c0823836fb5" containerID="4589b72cc516cd602e58285d5256bc2f52f89332fc9564290c0e8e45d9829aa2" exitCode=60
Oct 31 02:00:05.187134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.186505    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" event={"ID":"35bf319f-dfea-458f-be77-6c0823836fb5","Type":"ContainerDied","Data":"4589b72cc516cd602e58285d5256bc2f52f89332fc9564290c0e8e45d9829aa2"}
Oct 31 02:00:05.187134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.187079    5293 scope.go:122] "RemoveContainer" containerID="4589b72cc516cd602e58285d5256bc2f52f89332fc9564290c0e8e45d9829aa2"
Oct 31 02:00:05.307264 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.300697    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht" podStartSLOduration=5.300676 podStartE2EDuration="5.300676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:05.292338555 +0000 UTC m=+2363.240986949" watchObservedRunningTime="2026-10-31 02:00:05.300676 +0000 UTC m=+2363.249324394"
Oct 31 02:00:05.368315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:05.368264    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" podStartSLOduration=5.368251678 podStartE2EDuration="5.368251678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:05.363574135 +0000 UTC m=+2363.312222527" watchObservedRunningTime="2026-10-31 02:00:05.368251678 +0000 UTC m=+2363.316900064"
Oct 31 02:00:06.363846 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.362825    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" event={"ID":"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e","Type":"ContainerStarted","Data":"f786f4af7d03882eb11d7cbf87e2ce55d00442e415cc701f5c1fc4f446a5ed0d"}
Oct 31 02:00:06.366781 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.366755    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" event={"ID":"e5627e74-5b80-493a-bc75-2e42e13942cb","Type":"ContainerStarted","Data":"2c80c408519fa00084181718617d3b0c6744d1c4b7b95d2e902c05ec97680148"}
Oct 31 02:00:06.370222 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.370189    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" event={"ID":"a726d42d-1730-47aa-8a2d-1b453e4f4bab","Type":"ContainerStarted","Data":"4cc4393f97b3f8ca00775a02c1ecb43ab03dff747c4b37358477f82f0deb9f88"}
Oct 31 02:00:06.372318 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.372294    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" event={"ID":"aecd11ba-cc8a-48bb-8641-a0089eadbc07","Type":"ContainerStarted","Data":"bfd17cafacc45f04e2c1c003d59b49776d343767da13142340a0a2ce50b61177"}
Oct 31 02:00:06.376361 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.376331    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" event={"ID":"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a","Type":"ContainerStarted","Data":"dc47aaf05dd0f2cd6148bcdbccdf8811a4ff56cf887b8cece83b7c9b71a2a8cb"}
Oct 31 02:00:06.379405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.379387    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" event={"ID":"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660","Type":"ContainerStarted","Data":"2404abd64f26b79238c82805a76dd0bc05c801bc2885c5a3c19785b0c89bb2ea"}
Oct 31 02:00:06.380974 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.380956    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" event={"ID":"3681d17a-a7c0-45ac-9888-d44c117fc742","Type":"ContainerStarted","Data":"3432947e06541d4303b377e5b5aa187b1bcda5542700f7d083e3e2eac97d6c34"}
Oct 31 02:00:06.382342 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.382325    5293 generic.go:356] "Generic (PLEG): container finished" podID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerID="4768d0aaeab126e5713786ea2f22d489cb89636dbb49914eff59873d98dd7125" exitCode=60
Oct 31 02:00:06.382489 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.382404    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" event={"ID":"227a89a1-09f8-4819-8526-1340b1e8ca1d","Type":"ContainerDied","Data":"4768d0aaeab126e5713786ea2f22d489cb89636dbb49914eff59873d98dd7125"}
Oct 31 02:00:06.382612 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.382559    5293 scope.go:122] "RemoveContainer" containerID="4768d0aaeab126e5713786ea2f22d489cb89636dbb49914eff59873d98dd7125"
Oct 31 02:00:06.385690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.385651    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" podStartSLOduration=6.38563963 podStartE2EDuration="6.38563963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:06.382705393 +0000 UTC m=+2364.331353786" watchObservedRunningTime="2026-10-31 02:00:06.38563963 +0000 UTC m=+2364.334288021"
Oct 31 02:00:06.386535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.386511    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" event={"ID":"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6","Type":"ContainerStarted","Data":"2ffa9e5c2f45bc7702f16ee206e64c1a88102db2e59486e384edb7f30011dd9f"}
Oct 31 02:00:06.388820 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.388803    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" event={"ID":"35bf319f-dfea-458f-be77-6c0823836fb5","Type":"ContainerStarted","Data":"352f04566002a8f83742ec4ae0863bab604c5aff783c339b0d7206aa37d7fc32"}
Oct 31 02:00:06.483139 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.483081    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" podStartSLOduration=6.483062696 podStartE2EDuration="6.483062696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:06.481648737 +0000 UTC m=+2364.430297142" watchObservedRunningTime="2026-10-31 02:00:06.483062696 +0000 UTC m=+2364.431711088"
Oct 31 02:00:06.512948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.512891    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" podStartSLOduration=6.512872825 podStartE2EDuration="6.512872825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:06.500763827 +0000 UTC m=+2364.449412217" watchObservedRunningTime="2026-10-31 02:00:06.512872825 +0000 UTC m=+2364.461521220"
Oct 31 02:00:06.522586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:06.522519    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" podStartSLOduration=6.522170299 podStartE2EDuration="6.522170299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:06.522004711 +0000 UTC m=+2364.470653102" watchObservedRunningTime="2026-10-31 02:00:06.522170299 +0000 UTC m=+2364.470818692"
Oct 31 02:00:07.033460 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.033419    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"]
Oct 31 02:00:07.037924 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.037895    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934c6cf-fsdz7"]
Oct 31 02:00:07.401824 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.401789    5293 generic.go:356] "Generic (PLEG): container finished" podID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerID="2c80c408519fa00084181718617d3b0c6744d1c4b7b95d2e902c05ec97680148" exitCode=60
Oct 31 02:00:07.402390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.401876    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" event={"ID":"e5627e74-5b80-493a-bc75-2e42e13942cb","Type":"ContainerDied","Data":"2c80c408519fa00084181718617d3b0c6744d1c4b7b95d2e902c05ec97680148"}
Oct 31 02:00:07.402390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.401964    5293 scope.go:122] "RemoveContainer" containerID="c712ba8a99459d8d0daff8e7c5bc583c1b2a82f1ca6b431fb0252b4cfb249957"
Oct 31 02:00:07.402617 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.402596    5293 scope.go:122] "RemoveContainer" containerID="2c80c408519fa00084181718617d3b0c6744d1c4b7b95d2e902c05ec97680148"
Oct 31 02:00:07.403081 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.403057    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9_odh-ai-gateway-infra(e5627e74-5b80-493a-bc75-2e42e13942cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb"
Oct 31 02:00:07.405980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.405961    5293 generic.go:356] "Generic (PLEG): container finished" podID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerID="4cc4393f97b3f8ca00775a02c1ecb43ab03dff747c4b37358477f82f0deb9f88" exitCode=60
Oct 31 02:00:07.406106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.406077    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" event={"ID":"a726d42d-1730-47aa-8a2d-1b453e4f4bab","Type":"ContainerDied","Data":"4cc4393f97b3f8ca00775a02c1ecb43ab03dff747c4b37358477f82f0deb9f88"}
Oct 31 02:00:07.406387 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.406370    5293 scope.go:122] "RemoveContainer" containerID="4cc4393f97b3f8ca00775a02c1ecb43ab03dff747c4b37358477f82f0deb9f88"
Oct 31 02:00:07.406739 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.406716    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8_odh-ai-gateway-infra(a726d42d-1730-47aa-8a2d-1b453e4f4bab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab"
Oct 31 02:00:07.408942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.408916    5293 generic.go:356] "Generic (PLEG): container finished" podID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerID="bfd17cafacc45f04e2c1c003d59b49776d343767da13142340a0a2ce50b61177" exitCode=60
Oct 31 02:00:07.409111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.408951    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" event={"ID":"aecd11ba-cc8a-48bb-8641-a0089eadbc07","Type":"ContainerDied","Data":"bfd17cafacc45f04e2c1c003d59b49776d343767da13142340a0a2ce50b61177"}
Oct 31 02:00:07.409344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.409325    5293 scope.go:122] "RemoveContainer" containerID="bfd17cafacc45f04e2c1c003d59b49776d343767da13142340a0a2ce50b61177"
Oct 31 02:00:07.409742 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.409715    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm_odh-ai-gateway-infra(aecd11ba-cc8a-48bb-8641-a0089eadbc07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07"
Oct 31 02:00:07.412164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.412139    5293 generic.go:356] "Generic (PLEG): container finished" podID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerID="dc47aaf05dd0f2cd6148bcdbccdf8811a4ff56cf887b8cece83b7c9b71a2a8cb" exitCode=60
Oct 31 02:00:07.412610 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.412593    5293 scope.go:122] "RemoveContainer" containerID="dc47aaf05dd0f2cd6148bcdbccdf8811a4ff56cf887b8cece83b7c9b71a2a8cb"
Oct 31 02:00:07.412986 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.412965    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk_odh-ai-gateway-infra(460f3b5d-e0cf-45d2-bbe6-a468cd04f24a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a"
Oct 31 02:00:07.413034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.413014    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" event={"ID":"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a","Type":"ContainerDied","Data":"dc47aaf05dd0f2cd6148bcdbccdf8811a4ff56cf887b8cece83b7c9b71a2a8cb"}
Oct 31 02:00:07.415405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.415384    5293 generic.go:356] "Generic (PLEG): container finished" podID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerID="2404abd64f26b79238c82805a76dd0bc05c801bc2885c5a3c19785b0c89bb2ea" exitCode=60
Oct 31 02:00:07.415540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.415517    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" event={"ID":"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660","Type":"ContainerDied","Data":"2404abd64f26b79238c82805a76dd0bc05c801bc2885c5a3c19785b0c89bb2ea"}
Oct 31 02:00:07.415711 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.415691    5293 scope.go:122] "RemoveContainer" containerID="2404abd64f26b79238c82805a76dd0bc05c801bc2885c5a3c19785b0c89bb2ea"
Oct 31 02:00:07.416075 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.416051    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl_odh-ai-gateway-infra(ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660"
Oct 31 02:00:07.417661 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.417639    5293 generic.go:356] "Generic (PLEG): container finished" podID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerID="3432947e06541d4303b377e5b5aa187b1bcda5542700f7d083e3e2eac97d6c34" exitCode=60
Oct 31 02:00:07.417757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.417713    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" event={"ID":"3681d17a-a7c0-45ac-9888-d44c117fc742","Type":"ContainerDied","Data":"3432947e06541d4303b377e5b5aa187b1bcda5542700f7d083e3e2eac97d6c34"}
Oct 31 02:00:07.417954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.417940    5293 scope.go:122] "RemoveContainer" containerID="3432947e06541d4303b377e5b5aa187b1bcda5542700f7d083e3e2eac97d6c34"
Oct 31 02:00:07.418266 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.418245    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd_odh-ai-gateway-infra(3681d17a-a7c0-45ac-9888-d44c117fc742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742"
Oct 31 02:00:07.419990 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.419953    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" event={"ID":"227a89a1-09f8-4819-8526-1340b1e8ca1d","Type":"ContainerStarted","Data":"8b27e96fa42151a8fe79564e91e651052936eb1a8ba00b4d0aee25103e58bcf4"}
Oct 31 02:00:07.429372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.429324    5293 scope.go:122] "RemoveContainer" containerID="af422274289d0321d814427bb13a4b2fffddadc6ef7ee6598d5bebe69e227ed9"
Oct 31 02:00:07.429978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.429953    5293 generic.go:356] "Generic (PLEG): container finished" podID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerID="2ffa9e5c2f45bc7702f16ee206e64c1a88102db2e59486e384edb7f30011dd9f" exitCode=60
Oct 31 02:00:07.430309 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.430289    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" event={"ID":"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6","Type":"ContainerDied","Data":"2ffa9e5c2f45bc7702f16ee206e64c1a88102db2e59486e384edb7f30011dd9f"}
Oct 31 02:00:07.430798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.430779    5293 scope.go:122] "RemoveContainer" containerID="2ffa9e5c2f45bc7702f16ee206e64c1a88102db2e59486e384edb7f30011dd9f"
Oct 31 02:00:07.431252 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.431226    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd_odh-ai-gateway-infra(3aa20e9f-a3fd-4e8b-82a0-dba557a665d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6"
Oct 31 02:00:07.432500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.432479    5293 generic.go:356] "Generic (PLEG): container finished" podID="35bf319f-dfea-458f-be77-6c0823836fb5" containerID="352f04566002a8f83742ec4ae0863bab604c5aff783c339b0d7206aa37d7fc32" exitCode=60
Oct 31 02:00:07.432592 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.432559    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" event={"ID":"35bf319f-dfea-458f-be77-6c0823836fb5","Type":"ContainerDied","Data":"352f04566002a8f83742ec4ae0863bab604c5aff783c339b0d7206aa37d7fc32"}
Oct 31 02:00:07.432850 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.432833    5293 scope.go:122] "RemoveContainer" containerID="352f04566002a8f83742ec4ae0863bab604c5aff783c339b0d7206aa37d7fc32"
Oct 31 02:00:07.433213 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.433192    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-5ptbb_odh-ai-gateway-infra(35bf319f-dfea-458f-be77-6c0823836fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" podUID="35bf319f-dfea-458f-be77-6c0823836fb5"
Oct 31 02:00:07.434926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.434906    5293 generic.go:356] "Generic (PLEG): container finished" podID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerID="f786f4af7d03882eb11d7cbf87e2ce55d00442e415cc701f5c1fc4f446a5ed0d" exitCode=60
Oct 31 02:00:07.435097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.435074    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" event={"ID":"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e","Type":"ContainerDied","Data":"f786f4af7d03882eb11d7cbf87e2ce55d00442e415cc701f5c1fc4f446a5ed0d"}
Oct 31 02:00:07.435317 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.435300    5293 scope.go:122] "RemoveContainer" containerID="f786f4af7d03882eb11d7cbf87e2ce55d00442e415cc701f5c1fc4f446a5ed0d"
Oct 31 02:00:07.435681 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:07.435656    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt_odh-ai-gateway-infra(52eee9f7-0464-415f-ad3d-fbbdc4de3c5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e"
Oct 31 02:00:07.446133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.446080    5293 scope.go:122] "RemoveContainer" containerID="897eabf033e1548ba38c0acd079dac9d50536e3e3872447c8da7d5984ed9b1be"
Oct 31 02:00:07.461771 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.461754    5293 scope.go:122] "RemoveContainer" containerID="faa0cc1be8590f8bc68564da9b4abbd4f0e2c7db6b226021875a67dc74afcabe"
Oct 31 02:00:07.476171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.476129    5293 scope.go:122] "RemoveContainer" containerID="8abdbaa4a5c6805c300a61e35d70a49ba881f0af7cf66d3c93c975055f94464e"
Oct 31 02:00:07.491762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.491741    5293 scope.go:122] "RemoveContainer" containerID="b88630a9e94b2482f8954e94213162a006301825fe85330998bb023972e26648"
Oct 31 02:00:07.508413 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.508392    5293 scope.go:122] "RemoveContainer" containerID="db8c596df3465482aa31ce165028a0fcc892472a46c36547f239082fbc706863"
Oct 31 02:00:07.525197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.525170    5293 scope.go:122] "RemoveContainer" containerID="4589b72cc516cd602e58285d5256bc2f52f89332fc9564290c0e8e45d9829aa2"
Oct 31 02:00:07.539007 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:07.538988    5293 scope.go:122] "RemoveContainer" containerID="34f9f11148c88e82bf21ec2afa6911ae731e8ba48b490d01d5090b38ae65d7f5"
Oct 31 02:00:08.177650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.177618    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baaaabb2-1d46-4d9d-97ce-747cded4ae3a" path="/var/lib/kubelet/pods/baaaabb2-1d46-4d9d-97ce-747cded4ae3a/volumes"
Oct 31 02:00:08.449133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.449011    5293 scope.go:122] "RemoveContainer" containerID="2c80c408519fa00084181718617d3b0c6744d1c4b7b95d2e902c05ec97680148"
Oct 31 02:00:08.449564 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.449399    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9_odh-ai-gateway-infra(e5627e74-5b80-493a-bc75-2e42e13942cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb"
Oct 31 02:00:08.451143 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.451120    5293 scope.go:122] "RemoveContainer" containerID="4cc4393f97b3f8ca00775a02c1ecb43ab03dff747c4b37358477f82f0deb9f88"
Oct 31 02:00:08.451617 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.451591    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8_odh-ai-gateway-infra(a726d42d-1730-47aa-8a2d-1b453e4f4bab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab"
Oct 31 02:00:08.452902 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.452879    5293 scope.go:122] "RemoveContainer" containerID="bfd17cafacc45f04e2c1c003d59b49776d343767da13142340a0a2ce50b61177"
Oct 31 02:00:08.453217 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.453197    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm_odh-ai-gateway-infra(aecd11ba-cc8a-48bb-8641-a0089eadbc07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07"
Oct 31 02:00:08.454607 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.454563    5293 scope.go:122] "RemoveContainer" containerID="dc47aaf05dd0f2cd6148bcdbccdf8811a4ff56cf887b8cece83b7c9b71a2a8cb"
Oct 31 02:00:08.454847 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.454828    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk_odh-ai-gateway-infra(460f3b5d-e0cf-45d2-bbe6-a468cd04f24a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a"
Oct 31 02:00:08.456121 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.456104    5293 scope.go:122] "RemoveContainer" containerID="2404abd64f26b79238c82805a76dd0bc05c801bc2885c5a3c19785b0c89bb2ea"
Oct 31 02:00:08.456369 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.456347    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl_odh-ai-gateway-infra(ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660"
Oct 31 02:00:08.457706 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.457689    5293 scope.go:122] "RemoveContainer" containerID="3432947e06541d4303b377e5b5aa187b1bcda5542700f7d083e3e2eac97d6c34"
Oct 31 02:00:08.457973 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.457953    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd_odh-ai-gateway-infra(3681d17a-a7c0-45ac-9888-d44c117fc742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742"
Oct 31 02:00:08.459354 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.459335    5293 generic.go:356] "Generic (PLEG): container finished" podID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerID="8b27e96fa42151a8fe79564e91e651052936eb1a8ba00b4d0aee25103e58bcf4" exitCode=60
Oct 31 02:00:08.459450 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.459408    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" event={"ID":"227a89a1-09f8-4819-8526-1340b1e8ca1d","Type":"ContainerDied","Data":"8b27e96fa42151a8fe79564e91e651052936eb1a8ba00b4d0aee25103e58bcf4"}
Oct 31 02:00:08.459494 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.459456    5293 scope.go:122] "RemoveContainer" containerID="4768d0aaeab126e5713786ea2f22d489cb89636dbb49914eff59873d98dd7125"
Oct 31 02:00:08.459643 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.459625    5293 scope.go:122] "RemoveContainer" containerID="8b27e96fa42151a8fe79564e91e651052936eb1a8ba00b4d0aee25103e58bcf4"
Oct 31 02:00:08.459904 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.459882    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l_odh-ai-gateway-infra(227a89a1-09f8-4819-8526-1340b1e8ca1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d"
Oct 31 02:00:08.461442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.461419    5293 scope.go:122] "RemoveContainer" containerID="2ffa9e5c2f45bc7702f16ee206e64c1a88102db2e59486e384edb7f30011dd9f"
Oct 31 02:00:08.461671 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.461654    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd_odh-ai-gateway-infra(3aa20e9f-a3fd-4e8b-82a0-dba557a665d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6"
Oct 31 02:00:08.462974 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.462956    5293 scope.go:122] "RemoveContainer" containerID="352f04566002a8f83742ec4ae0863bab604c5aff783c339b0d7206aa37d7fc32"
Oct 31 02:00:08.463278 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.463255    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-5ptbb_odh-ai-gateway-infra(35bf319f-dfea-458f-be77-6c0823836fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" podUID="35bf319f-dfea-458f-be77-6c0823836fb5"
Oct 31 02:00:08.464625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:08.464608    5293 scope.go:122] "RemoveContainer" containerID="f786f4af7d03882eb11d7cbf87e2ce55d00442e415cc701f5c1fc4f446a5ed0d"
Oct 31 02:00:08.464816 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:08.464798    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt_odh-ai-gateway-infra(52eee9f7-0464-415f-ad3d-fbbdc4de3c5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e"
Oct 31 02:00:09.130154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.129264    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"]
Oct 31 02:00:09.130344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.130329    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852" containerName="revoke-keys"
Oct 31 02:00:09.143190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.142454    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.144117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.144087    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"]
Oct 31 02:00:09.147521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.147495    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d0c6ff-openshift-default-dockercfg-j64m7\""
Oct 31 02:00:09.187322 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.187287    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.187535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.187376    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtc7s\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-kube-api-access-rtc7s\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.190173 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.190144    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.190292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.190199    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.190292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.190258    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.190292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.190274    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaac94e-52de-410f-a032-7615ba6c9992-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.190485 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.190324    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.190485 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.190388    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaac94e-52de-410f-a032-7615ba6c9992-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.190485 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.190424    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.303069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.294126    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.316807    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.316861    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtc7s\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-kube-api-access-rtc7s\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.316995    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.317095    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.317217    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.317243    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaac94e-52de-410f-a032-7615ba6c9992-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.317351    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.317439    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaac94e-52de-410f-a032-7615ba6c9992-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.317512    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.321101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.317953    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.376279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.362997    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.376279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.363305    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.376279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.364607    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaac94e-52de-410f-a032-7615ba6c9992-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.432079 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.424775    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtc7s\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-kube-api-access-rtc7s\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.434448 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.433011    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaac94e-52de-410f-a032-7615ba6c9992-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.449088 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.434837    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.460641 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.455961    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:09.517413 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.516279    5293 scope.go:122] "RemoveContainer" containerID="8b27e96fa42151a8fe79564e91e651052936eb1a8ba00b4d0aee25103e58bcf4"
Oct 31 02:00:09.517413 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:09.516764    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l_odh-ai-gateway-infra(227a89a1-09f8-4819-8526-1340b1e8ca1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d"
Oct 31 02:00:09.761149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:09.761066    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:10.295063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:10.289153    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:00:10.295063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:10.290164    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:00:10.295063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:10.290265    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:00:10.340904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:10.340868    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"]
Oct 31 02:00:10.537493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:10.537453    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" event={"ID":"1eaac94e-52de-410f-a032-7615ba6c9992","Type":"ContainerStarted","Data":"47b9d1b280a322409f020b4c086ca1cc1931dbb2d6f85f7669130a465ba65f27"}
Oct 31 02:00:11.092366 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:11.092323    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:00:11.556263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:11.556229    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" event={"ID":"1eaac94e-52de-410f-a032-7615ba6c9992","Type":"ContainerStarted","Data":"1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad"}
Oct 31 02:00:11.594460 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:11.594412    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" podStartSLOduration=2.5943995600000003 podStartE2EDuration="2.59439956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:11.589518505 +0000 UTC m=+2369.538166921" watchObservedRunningTime="2026-10-31 02:00:11.59439956 +0000 UTC m=+2369.543047985"
Oct 31 02:00:11.781062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:11.768397    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:11.805137 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:11.804571    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:11.882513 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:11.882435    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:00:12.052024 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:12.051995    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"]
Oct 31 02:00:12.059426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:12.059354    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8caac12-wjbfk"]
Oct 31 02:00:12.176374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:12.176344    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ee5480-48dd-4dbb-b146-e7c4ba29cca6" path="/var/lib/kubelet/pods/c5ee5480-48dd-4dbb-b146-e7c4ba29cca6/volumes"
Oct 31 02:00:12.565449 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:12.565368    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:12.566339 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:12.566312    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:00:15.592298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:15.575312    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"]
Oct 31 02:00:15.765065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:15.756410    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:15.765065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:15.760839    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:00:15.863127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:15.809870    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"]
Oct 31 02:00:15.921385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:15.918727    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:15.938948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:15.937189    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:15.940237 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:15.939247    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcm8\" (UniqueName: \"kubernetes.io/projected/9f05d658-ef99-4344-b7a0-1d691c708cca-kube-api-access-kgcm8\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:16.025063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.024833    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"]
Oct 31 02:00:16.057064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.047146    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065221    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065290    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nt4\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-kube-api-access-l9nt4\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065319    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcm8\" (UniqueName: \"kubernetes.io/projected/9f05d658-ef99-4344-b7a0-1d691c708cca-kube-api-access-kgcm8\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065347    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-data\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065409    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-certs\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065437    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-token\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065475    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065582    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065701    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065726    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-podinfo\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065780    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-envoy\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.065842    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-credential-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.068064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.066925    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:16.070852 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.069813    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2e6bfd-openshift-default-dockercfg-znd2f\""
Oct 31 02:00:16.097121 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.082152    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"]
Oct 31 02:00:16.111526 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.111495    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:16.125315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.120884    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcm8\" (UniqueName: \"kubernetes.io/projected/9f05d658-ef99-4344-b7a0-1d691c708cca-kube-api-access-kgcm8\") pod \"maas-api-revoke-sub-fe5762d9-kkhq6\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:16.166806 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.166773    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.166946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.166812    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-podinfo\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.166946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.166847    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-envoy\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.166946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.166879    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-credential-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.166993    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nt4\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-kube-api-access-l9nt4\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167034    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-data\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167119    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-certs\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167155    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-token\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167340 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167297    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167354    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-credential-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167515 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167491    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167589 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167565    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-data\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167673 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167651    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-certs\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.167730 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.167694    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.169629 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.169598    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-envoy\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.169629 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.169614    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-podinfo\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.181726 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.181700    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-token\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.184420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.184400    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nt4\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-kube-api-access-l9nt4\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.391512 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.391435    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:00:16.420081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.420053    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:16.452207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.452158    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:16.738510 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.738483    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"]
Oct 31 02:00:16.747650 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:00:16.747483    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f05d658_ef99_4344_b7a0_1d691c708cca.slice/crio-3c83212244b89c9b796d5e12d6ef2736cb54d72f1f068a34eb604f21a7617949 WatchSource:0}: Error finding container 3c83212244b89c9b796d5e12d6ef2736cb54d72f1f068a34eb604f21a7617949: Status 404 returned error can't find the container with id 3c83212244b89c9b796d5e12d6ef2736cb54d72f1f068a34eb604f21a7617949
Oct 31 02:00:16.866970 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.866939    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"]
Oct 31 02:00:16.867399 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:00:16.867372    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f02e10_5591_47c3_be44_5f9d7a3fc78f.slice/crio-648a11f45a031b7c385ff812c30d41c90b6a7a0d2837a37be29ee2816bcf0c89 WatchSource:0}: Error finding container 648a11f45a031b7c385ff812c30d41c90b6a7a0d2837a37be29ee2816bcf0c89: Status 404 returned error can't find the container with id 648a11f45a031b7c385ff812c30d41c90b6a7a0d2837a37be29ee2816bcf0c89
Oct 31 02:00:16.870420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.870383    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:00:16.870527 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.870462    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:00:16.870527 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:16.870506    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:00:17.108064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:17.108013    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6" event={"ID":"9f05d658-ef99-4344-b7a0-1d691c708cca","Type":"ContainerStarted","Data":"3c83212244b89c9b796d5e12d6ef2736cb54d72f1f068a34eb604f21a7617949"}
Oct 31 02:00:17.113750 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:17.113725    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" event={"ID":"d5f02e10-5591-47c3-be44-5f9d7a3fc78f","Type":"ContainerStarted","Data":"02bfbf1726a2eecb1b335896818cdfc46c1d8e5091113a135575ef92fc0da4fa"}
Oct 31 02:00:17.113968 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:17.113948    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" event={"ID":"d5f02e10-5591-47c3-be44-5f9d7a3fc78f","Type":"ContainerStarted","Data":"648a11f45a031b7c385ff812c30d41c90b6a7a0d2837a37be29ee2816bcf0c89"}
Oct 31 02:00:17.151592 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:17.151541    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podStartSLOduration=2.151524559 podStartE2EDuration="2.151524559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:17.148252702 +0000 UTC m=+2375.096901094" watchObservedRunningTime="2026-10-31 02:00:17.151524559 +0000 UTC m=+2375.100172952"
Oct 31 02:00:17.166945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:17.166921    5293 scope.go:122] "RemoveContainer" containerID="bfd17cafacc45f04e2c1c003d59b49776d343767da13142340a0a2ce50b61177"
Oct 31 02:00:17.462888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:17.460534    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:17.473751 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:17.473194    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 02:00:17.473751 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:17.473271    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 02:00:18.291847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:18.291802    5293 scope.go:122] "RemoveContainer" containerID="4cc4393f97b3f8ca00775a02c1ecb43ab03dff747c4b37358477f82f0deb9f88"
Oct 31 02:00:18.368435 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:18.366489    5293 scope.go:122] "RemoveContainer" containerID="2c80c408519fa00084181718617d3b0c6744d1c4b7b95d2e902c05ec97680148"
Oct 31 02:00:18.368435 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:18.368070    5293 scope.go:122] "RemoveContainer" containerID="f786f4af7d03882eb11d7cbf87e2ce55d00442e415cc701f5c1fc4f446a5ed0d"
Oct 31 02:00:18.389280 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:18.389253    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" event={"ID":"aecd11ba-cc8a-48bb-8641-a0089eadbc07","Type":"ContainerStarted","Data":"1b6d368e5ccb81cb13bfccfcfc407cabaa7c9a94437e0070a6b78f37ff6ae2a7"}
Oct 31 02:00:18.394471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:18.394445    5293 generic.go:356] "Generic (PLEG): container finished" podID="9f05d658-ef99-4344-b7a0-1d691c708cca" containerID="7ae5ba175adb8bcc8f31668ec4a303107761a61386e0f65c60bc2f0d890c5185" exitCode=0
Oct 31 02:00:18.394892 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:18.394863    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6" event={"ID":"9f05d658-ef99-4344-b7a0-1d691c708cca","Type":"ContainerDied","Data":"7ae5ba175adb8bcc8f31668ec4a303107761a61386e0f65c60bc2f0d890c5185"}
Oct 31 02:00:18.459149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:18.459114    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 02:00:18.459302 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:18.459182    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 02:00:19.166509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.166480    5293 scope.go:122] "RemoveContainer" containerID="2404abd64f26b79238c82805a76dd0bc05c801bc2885c5a3c19785b0c89bb2ea"
Oct 31 02:00:19.166660 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.166605    5293 scope.go:122] "RemoveContainer" containerID="2ffa9e5c2f45bc7702f16ee206e64c1a88102db2e59486e384edb7f30011dd9f"
Oct 31 02:00:19.444802 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.444342    5293 generic.go:356] "Generic (PLEG): container finished" podID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerID="1b6d368e5ccb81cb13bfccfcfc407cabaa7c9a94437e0070a6b78f37ff6ae2a7" exitCode=60
Oct 31 02:00:19.447319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.445212    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" event={"ID":"aecd11ba-cc8a-48bb-8641-a0089eadbc07","Type":"ContainerDied","Data":"1b6d368e5ccb81cb13bfccfcfc407cabaa7c9a94437e0070a6b78f37ff6ae2a7"}
Oct 31 02:00:19.447319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.446114    5293 scope.go:122] "RemoveContainer" containerID="bfd17cafacc45f04e2c1c003d59b49776d343767da13142340a0a2ce50b61177"
Oct 31 02:00:19.447319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.446752    5293 scope.go:122] "RemoveContainer" containerID="1b6d368e5ccb81cb13bfccfcfc407cabaa7c9a94437e0070a6b78f37ff6ae2a7"
Oct 31 02:00:19.447319 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:19.447256    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm_odh-ai-gateway-infra(aecd11ba-cc8a-48bb-8641-a0089eadbc07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07"
Oct 31 02:00:19.447660 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.447485    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"]
Oct 31 02:00:19.457462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.457431    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 02:00:19.457596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.457495    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 02:00:19.459204 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.459176    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" event={"ID":"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e","Type":"ContainerStarted","Data":"0ddafe16ba7cd7cd121500469f71f59a879807ab3f2644c0028d4844ec338856"}
Oct 31 02:00:19.463068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.463022    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" event={"ID":"e5627e74-5b80-493a-bc75-2e42e13942cb","Type":"ContainerStarted","Data":"b5d6cab64971944cade0cf4391beb24d68b6c4ce5fe7cd7594496ad03d22ab95"}
Oct 31 02:00:19.469908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:19.469855    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" event={"ID":"a726d42d-1730-47aa-8a2d-1b453e4f4bab","Type":"ContainerStarted","Data":"2fefd0a83e6243fd3d28aedafe031eff84ecda87e31319b33baa7e92bca8bf8e"}
Oct 31 02:00:20.166587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.166558    5293 scope.go:122] "RemoveContainer" containerID="dc47aaf05dd0f2cd6148bcdbccdf8811a4ff56cf887b8cece83b7c9b71a2a8cb"
Oct 31 02:00:20.166753 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.166731    5293 scope.go:122] "RemoveContainer" containerID="352f04566002a8f83742ec4ae0863bab604c5aff783c339b0d7206aa37d7fc32"
Oct 31 02:00:20.167142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.167118    5293 scope.go:122] "RemoveContainer" containerID="3432947e06541d4303b377e5b5aa187b1bcda5542700f7d083e3e2eac97d6c34"
Oct 31 02:00:20.249729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.249662    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:20.359504 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.359414    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f05d658-ef99-4344-b7a0-1d691c708cca-kube-api-access-kgcm8\" (UniqueName: \"kubernetes.io/projected/9f05d658-ef99-4344-b7a0-1d691c708cca-kube-api-access-kgcm8\") pod \"9f05d658-ef99-4344-b7a0-1d691c708cca\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") "
Oct 31 02:00:20.360894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.360859    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-openshift-ca\") pod \"9f05d658-ef99-4344-b7a0-1d691c708cca\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") "
Oct 31 02:00:20.361289 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.361259    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-openshift-ca" pod "9f05d658-ef99-4344-b7a0-1d691c708cca" (UID: "9f05d658-ef99-4344-b7a0-1d691c708cca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:20.361518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.361495    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-xks-ca\") pod \"9f05d658-ef99-4344-b7a0-1d691c708cca\" (UID: \"9f05d658-ef99-4344-b7a0-1d691c708cca\") "
Oct 31 02:00:20.362904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.362781    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.364155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.364108    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f05d658-ef99-4344-b7a0-1d691c708cca-kube-api-access-kgcm8" pod "9f05d658-ef99-4344-b7a0-1d691c708cca" (UID: "9f05d658-ef99-4344-b7a0-1d691c708cca"). InnerVolumeSpecName "kube-api-access-kgcm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:20.364949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.364911    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-xks-ca" pod "9f05d658-ef99-4344-b7a0-1d691c708cca" (UID: "9f05d658-ef99-4344-b7a0-1d691c708cca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:00:20.457784 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.457754    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 02:00:20.458199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.457807    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 02:00:20.465207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.465182    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f05d658-ef99-4344-b7a0-1d691c708cca-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.465207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.465207    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcm8\" (UniqueName: \"kubernetes.io/projected/9f05d658-ef99-4344-b7a0-1d691c708cca-kube-api-access-kgcm8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.476434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.476408    5293 generic.go:356] "Generic (PLEG): container finished" podID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerID="b5d6cab64971944cade0cf4391beb24d68b6c4ce5fe7cd7594496ad03d22ab95" exitCode=60
Oct 31 02:00:20.476558 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.476484    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" event={"ID":"e5627e74-5b80-493a-bc75-2e42e13942cb","Type":"ContainerDied","Data":"b5d6cab64971944cade0cf4391beb24d68b6c4ce5fe7cd7594496ad03d22ab95"}
Oct 31 02:00:20.476626 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.476561    5293 scope.go:122] "RemoveContainer" containerID="2c80c408519fa00084181718617d3b0c6744d1c4b7b95d2e902c05ec97680148"
Oct 31 02:00:20.476957 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.476938    5293 scope.go:122] "RemoveContainer" containerID="b5d6cab64971944cade0cf4391beb24d68b6c4ce5fe7cd7594496ad03d22ab95"
Oct 31 02:00:20.477422 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:20.477400    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9_odh-ai-gateway-infra(e5627e74-5b80-493a-bc75-2e42e13942cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb"
Oct 31 02:00:20.478891 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.478870    5293 generic.go:356] "Generic (PLEG): container finished" podID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerID="2fefd0a83e6243fd3d28aedafe031eff84ecda87e31319b33baa7e92bca8bf8e" exitCode=60
Oct 31 02:00:20.478998 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.478953    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" event={"ID":"a726d42d-1730-47aa-8a2d-1b453e4f4bab","Type":"ContainerDied","Data":"2fefd0a83e6243fd3d28aedafe031eff84ecda87e31319b33baa7e92bca8bf8e"}
Oct 31 02:00:20.479380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.479363    5293 scope.go:122] "RemoveContainer" containerID="2fefd0a83e6243fd3d28aedafe031eff84ecda87e31319b33baa7e92bca8bf8e"
Oct 31 02:00:20.479752 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:20.479729    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8_odh-ai-gateway-infra(a726d42d-1730-47aa-8a2d-1b453e4f4bab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab"
Oct 31 02:00:20.482991 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.482959    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" event={"ID":"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660","Type":"ContainerStarted","Data":"a5c1708bcc9e4a30289f53deb518d6d24899a3db1fa758c88634eed9d3ff93e0"}
Oct 31 02:00:20.485219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.485190    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" event={"ID":"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6","Type":"ContainerStarted","Data":"aa404ec3c4350b1ecd5ca6bdb53c794ddef1114efda9f15c8574937326780576"}
Oct 31 02:00:20.487078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.487055    5293 generic.go:356] "Generic (PLEG): container finished" podID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerID="0ddafe16ba7cd7cd121500469f71f59a879807ab3f2644c0028d4844ec338856" exitCode=60
Oct 31 02:00:20.487078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.487067    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" event={"ID":"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e","Type":"ContainerDied","Data":"0ddafe16ba7cd7cd121500469f71f59a879807ab3f2644c0028d4844ec338856"}
Oct 31 02:00:20.487437 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.487419    5293 scope.go:122] "RemoveContainer" containerID="0ddafe16ba7cd7cd121500469f71f59a879807ab3f2644c0028d4844ec338856"
Oct 31 02:00:20.487754 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:20.487731    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt_odh-ai-gateway-infra(52eee9f7-0464-415f-ad3d-fbbdc4de3c5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e"
Oct 31 02:00:20.488669 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.488654    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"
Oct 31 02:00:20.488669 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.488659    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6" event={"ID":"9f05d658-ef99-4344-b7a0-1d691c708cca","Type":"ContainerDied","Data":"3c83212244b89c9b796d5e12d6ef2736cb54d72f1f068a34eb604f21a7617949"}
Oct 31 02:00:20.488842 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.488698    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c83212244b89c9b796d5e12d6ef2736cb54d72f1f068a34eb604f21a7617949"
Oct 31 02:00:20.493979 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.493959    5293 scope.go:122] "RemoveContainer" containerID="4cc4393f97b3f8ca00775a02c1ecb43ab03dff747c4b37358477f82f0deb9f88"
Oct 31 02:00:20.716376 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.663566    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"]
Oct 31 02:00:20.858540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.858095    5293 scope.go:122] "RemoveContainer" containerID="f786f4af7d03882eb11d7cbf87e2ce55d00442e415cc701f5c1fc4f446a5ed0d"
Oct 31 02:00:20.858540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.858251    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"]
Oct 31 02:00:20.885296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:20.882105    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"]
Oct 31 02:00:21.046990 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.046907    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"]
Oct 31 02:00:21.053320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.053290    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b7f0b1-wtwjq"]
Oct 31 02:00:21.084139 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.084113    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:21.165259 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.165226    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecd11ba-cc8a-48bb-8641-a0089eadbc07-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecd11ba-cc8a-48bb-8641-a0089eadbc07-maas-api-service-ca\") pod \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\" (UID: \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\") "
Oct 31 02:00:21.165445 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.165421    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecd11ba-cc8a-48bb-8641-a0089eadbc07-kube-api-access-zc7dv\" (UniqueName: \"kubernetes.io/projected/aecd11ba-cc8a-48bb-8641-a0089eadbc07-kube-api-access-zc7dv\") pod \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\" (UID: \"aecd11ba-cc8a-48bb-8641-a0089eadbc07\") "
Oct 31 02:00:21.165585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.165562    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecd11ba-cc8a-48bb-8641-a0089eadbc07-maas-api-service-ca" pod "aecd11ba-cc8a-48bb-8641-a0089eadbc07" (UID: "aecd11ba-cc8a-48bb-8641-a0089eadbc07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:21.165935 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.165918    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aecd11ba-cc8a-48bb-8641-a0089eadbc07-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.166533 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.166513    5293 scope.go:122] "RemoveContainer" containerID="8b27e96fa42151a8fe79564e91e651052936eb1a8ba00b4d0aee25103e58bcf4"
Oct 31 02:00:21.167772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.167748    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecd11ba-cc8a-48bb-8641-a0089eadbc07-kube-api-access-zc7dv" pod "aecd11ba-cc8a-48bb-8641-a0089eadbc07" (UID: "aecd11ba-cc8a-48bb-8641-a0089eadbc07"). InnerVolumeSpecName "kube-api-access-zc7dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:21.268255 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.268071    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc7dv\" (UniqueName: \"kubernetes.io/projected/aecd11ba-cc8a-48bb-8641-a0089eadbc07-kube-api-access-zc7dv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.493092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.487445    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 02:00:21.493092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.487506    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 02:00:21.569874 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.569840    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm" event={"ID":"aecd11ba-cc8a-48bb-8641-a0089eadbc07","Type":"ContainerDied","Data":"fcf45653bc58ea8c1ba7b4686089d916f32dac12ab1fb13dfcb54d9253e6448a"}
Oct 31 02:00:21.570029 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.570018    5293 scope.go:122] "RemoveContainer" containerID="1b6d368e5ccb81cb13bfccfcfc407cabaa7c9a94437e0070a6b78f37ff6ae2a7"
Oct 31 02:00:21.570161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.570142    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"
Oct 31 02:00:21.583197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.583167    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" event={"ID":"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a","Type":"ContainerStarted","Data":"e0949f60c5957326414f107a066b13fb3e31ca0c1cb6a8d9b16207cf1e309d8e"}
Oct 31 02:00:21.583634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.583614    5293 scope.go:122] "RemoveContainer" containerID="e0949f60c5957326414f107a066b13fb3e31ca0c1cb6a8d9b16207cf1e309d8e"
Oct 31 02:00:21.583991 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:21.583964    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk_odh-ai-gateway-infra(460f3b5d-e0cf-45d2-bbe6-a468cd04f24a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a"
Oct 31 02:00:21.591554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.591528    5293 generic.go:356] "Generic (PLEG): container finished" podID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerID="a5c1708bcc9e4a30289f53deb518d6d24899a3db1fa758c88634eed9d3ff93e0" exitCode=60
Oct 31 02:00:21.592427 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.591718    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" event={"ID":"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660","Type":"ContainerDied","Data":"a5c1708bcc9e4a30289f53deb518d6d24899a3db1fa758c88634eed9d3ff93e0"}
Oct 31 02:00:21.592427 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.592287    5293 scope.go:122] "RemoveContainer" containerID="a5c1708bcc9e4a30289f53deb518d6d24899a3db1fa758c88634eed9d3ff93e0"
Oct 31 02:00:21.593110 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:21.592902    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl_odh-ai-gateway-infra(ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660"
Oct 31 02:00:21.649165 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.648988    5293 scope.go:122] "RemoveContainer" containerID="2404abd64f26b79238c82805a76dd0bc05c801bc2885c5a3c19785b0c89bb2ea"
Oct 31 02:00:21.681569 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.681536    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" event={"ID":"3681d17a-a7c0-45ac-9888-d44c117fc742","Type":"ContainerStarted","Data":"d4f275d5d13088271f15e5341cbf848f03ad5b9038e921027c63793142d39c80"}
Oct 31 02:00:21.685272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.685247    5293 generic.go:356] "Generic (PLEG): container finished" podID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerID="aa404ec3c4350b1ecd5ca6bdb53c794ddef1114efda9f15c8574937326780576" exitCode=60
Oct 31 02:00:21.685492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.685474    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" event={"ID":"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6","Type":"ContainerDied","Data":"aa404ec3c4350b1ecd5ca6bdb53c794ddef1114efda9f15c8574937326780576"}
Oct 31 02:00:21.708828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.695353    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" event={"ID":"35bf319f-dfea-458f-be77-6c0823836fb5","Type":"ContainerStarted","Data":"0b862fc9e63e08b35483c3085fdfbe621edec233d8742b1538a63b3609712a6e"}
Oct 31 02:00:21.709344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.709317    5293 scope.go:122] "RemoveContainer" containerID="aa404ec3c4350b1ecd5ca6bdb53c794ddef1114efda9f15c8574937326780576"
Oct 31 02:00:21.720156 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:21.710140    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd_odh-ai-gateway-infra(3aa20e9f-a3fd-4e8b-82a0-dba557a665d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6"
Oct 31 02:00:21.764116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.721238    5293 scope.go:122] "RemoveContainer" containerID="0b862fc9e63e08b35483c3085fdfbe621edec233d8742b1538a63b3609712a6e"
Oct 31 02:00:21.767270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.726305    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"]
Oct 31 02:00:21.767270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:21.767002    5293 scope.go:122] "RemoveContainer" containerID="2ffa9e5c2f45bc7702f16ee206e64c1a88102db2e59486e384edb7f30011dd9f"
Oct 31 02:00:21.767653 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:21.767615    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-5ptbb_odh-ai-gateway-infra(35bf319f-dfea-458f-be77-6c0823836fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" podUID="35bf319f-dfea-458f-be77-6c0823836fb5"
Oct 31 02:00:22.523064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.477250    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 02:00:22.554633 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.554579    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 02:00:22.781390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.781351    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:00:22.878139 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.877986    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99752b74-8d5c-4304-b904-49420758454b" path="/var/lib/kubelet/pods/99752b74-8d5c-4304-b904-49420758454b/volumes"
Oct 31 02:00:22.878935 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.878831    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"]
Oct 31 02:00:22.884804 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.884776    5293 generic.go:356] "Generic (PLEG): container finished" podID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerID="d4f275d5d13088271f15e5341cbf848f03ad5b9038e921027c63793142d39c80" exitCode=60
Oct 31 02:00:22.884928 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.884804    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" event={"ID":"3681d17a-a7c0-45ac-9888-d44c117fc742","Type":"ContainerDied","Data":"d4f275d5d13088271f15e5341cbf848f03ad5b9038e921027c63793142d39c80"}
Oct 31 02:00:22.884928 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.884858    5293 scope.go:122] "RemoveContainer" containerID="3432947e06541d4303b377e5b5aa187b1bcda5542700f7d083e3e2eac97d6c34"
Oct 31 02:00:22.885300 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.885286    5293 scope.go:122] "RemoveContainer" containerID="d4f275d5d13088271f15e5341cbf848f03ad5b9038e921027c63793142d39c80"
Oct 31 02:00:22.885629 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:22.885608    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd_odh-ai-gateway-infra(3681d17a-a7c0-45ac-9888-d44c117fc742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742"
Oct 31 02:00:22.888614 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.888597    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:22.889209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.889191    5293 generic.go:356] "Generic (PLEG): container finished" podID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerID="a578019f13cc691e4b24bf30214f6086e82dd97409f1d29104ed107aef50648c" exitCode=60
Oct 31 02:00:22.889373 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.889248    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" event={"ID":"227a89a1-09f8-4819-8526-1340b1e8ca1d","Type":"ContainerDied","Data":"a578019f13cc691e4b24bf30214f6086e82dd97409f1d29104ed107aef50648c"}
Oct 31 02:00:22.889842 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.889809    5293 scope.go:122] "RemoveContainer" containerID="a578019f13cc691e4b24bf30214f6086e82dd97409f1d29104ed107aef50648c"
Oct 31 02:00:22.890088 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.890032    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"]
Oct 31 02:00:22.890222 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:22.890201    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l_odh-ai-gateway-infra(227a89a1-09f8-4819-8526-1340b1e8ca1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d"
Oct 31 02:00:22.891541 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.891523    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.895727    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" event={"ID":"35bf319f-dfea-458f-be77-6c0823836fb5","Type":"ContainerDied","Data":"0b862fc9e63e08b35483c3085fdfbe621edec233d8742b1538a63b3609712a6e"}
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.895920    5293 generic.go:356] "Generic (PLEG): container finished" podID="35bf319f-dfea-458f-be77-6c0823836fb5" containerID="0b862fc9e63e08b35483c3085fdfbe621edec233d8742b1538a63b3609712a6e" exitCode=60
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.896034    5293 scope.go:122] "RemoveContainer" containerID="0b862fc9e63e08b35483c3085fdfbe621edec233d8742b1538a63b3609712a6e"
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.896094    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890200-jc8wm"]
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:22.896374    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-5ptbb_odh-ai-gateway-infra(35bf319f-dfea-458f-be77-6c0823836fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" podUID="35bf319f-dfea-458f-be77-6c0823836fb5"
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.896459    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.899652    5293 generic.go:356] "Generic (PLEG): container finished" podID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerID="e0949f60c5957326414f107a066b13fb3e31ca0c1cb6a8d9b16207cf1e309d8e" exitCode=60
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.899723    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" event={"ID":"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a","Type":"ContainerDied","Data":"e0949f60c5957326414f107a066b13fb3e31ca0c1cb6a8d9b16207cf1e309d8e"}
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.900120    5293 scope.go:122] "RemoveContainer" containerID="e0949f60c5957326414f107a066b13fb3e31ca0c1cb6a8d9b16207cf1e309d8e"
Oct 31 02:00:22.927158 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:22.900432    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk_odh-ai-gateway-infra(460f3b5d-e0cf-45d2-bbe6-a468cd04f24a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a"
Oct 31 02:00:22.950673 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.950644    5293 scope.go:122] "RemoveContainer" containerID="8b27e96fa42151a8fe79564e91e651052936eb1a8ba00b4d0aee25103e58bcf4"
Oct 31 02:00:22.973102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.969333    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a726d42d-1730-47aa-8a2d-1b453e4f4bab-kube-api-access-4c48l\" (UniqueName: \"kubernetes.io/projected/a726d42d-1730-47aa-8a2d-1b453e4f4bab-kube-api-access-4c48l\") pod \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\" (UID: \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\") "
Oct 31 02:00:22.973102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.969427    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5627e74-5b80-493a-bc75-2e42e13942cb-kube-api-access-784qq\" (UniqueName: \"kubernetes.io/projected/e5627e74-5b80-493a-bc75-2e42e13942cb-kube-api-access-784qq\") pod \"e5627e74-5b80-493a-bc75-2e42e13942cb\" (UID: \"e5627e74-5b80-493a-bc75-2e42e13942cb\") "
Oct 31 02:00:22.973102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.969468    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a726d42d-1730-47aa-8a2d-1b453e4f4bab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a726d42d-1730-47aa-8a2d-1b453e4f4bab-maas-api-service-ca\") pod \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\" (UID: \"a726d42d-1730-47aa-8a2d-1b453e4f4bab\") "
Oct 31 02:00:22.973102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.971993    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a726d42d-1730-47aa-8a2d-1b453e4f4bab-maas-api-service-ca" pod "a726d42d-1730-47aa-8a2d-1b453e4f4bab" (UID: "a726d42d-1730-47aa-8a2d-1b453e4f4bab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:22.981403 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.981256    5293 scope.go:122] "RemoveContainer" containerID="352f04566002a8f83742ec4ae0863bab604c5aff783c339b0d7206aa37d7fc32"
Oct 31 02:00:22.984702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.984471    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a726d42d-1730-47aa-8a2d-1b453e4f4bab-kube-api-access-4c48l" pod "a726d42d-1730-47aa-8a2d-1b453e4f4bab" (UID: "a726d42d-1730-47aa-8a2d-1b453e4f4bab"). InnerVolumeSpecName "kube-api-access-4c48l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:22.984874 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:22.984728    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5627e74-5b80-493a-bc75-2e42e13942cb-kube-api-access-784qq" pod "e5627e74-5b80-493a-bc75-2e42e13942cb" (UID: "e5627e74-5b80-493a-bc75-2e42e13942cb"). InnerVolumeSpecName "kube-api-access-784qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:23.058562 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.057803    5293 scope.go:122] "RemoveContainer" containerID="dc47aaf05dd0f2cd6148bcdbccdf8811a4ff56cf887b8cece83b7c9b71a2a8cb"
Oct 31 02:00:23.062671 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.062641    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"]
Oct 31 02:00:23.070927 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.070904    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-maas-api-service-ca\") pod \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\" (UID: \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\") "
Oct 31 02:00:23.071419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.071367    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-maas-api-service-ca" pod "52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" (UID: "52eee9f7-0464-415f-ad3d-fbbdc4de3c5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:23.071620 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.071598    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5627e74-5b80-493a-bc75-2e42e13942cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5627e74-5b80-493a-bc75-2e42e13942cb-maas-api-service-ca\") pod \"e5627e74-5b80-493a-bc75-2e42e13942cb\" (UID: \"e5627e74-5b80-493a-bc75-2e42e13942cb\") "
Oct 31 02:00:23.073006 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.071950    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5627e74-5b80-493a-bc75-2e42e13942cb-maas-api-service-ca" pod "e5627e74-5b80-493a-bc75-2e42e13942cb" (UID: "e5627e74-5b80-493a-bc75-2e42e13942cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:23.073206 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.073190    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-kube-api-access-pldgj\" (UniqueName: \"kubernetes.io/projected/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-kube-api-access-pldgj\") pod \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\" (UID: \"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e\") "
Oct 31 02:00:23.074353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.074333    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c48l\" (UniqueName: \"kubernetes.io/projected/a726d42d-1730-47aa-8a2d-1b453e4f4bab-kube-api-access-4c48l\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.074464 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.074452    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-784qq\" (UniqueName: \"kubernetes.io/projected/e5627e74-5b80-493a-bc75-2e42e13942cb-kube-api-access-784qq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.074539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.074529    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a726d42d-1730-47aa-8a2d-1b453e4f4bab-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.074609 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.074599    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5627e74-5b80-493a-bc75-2e42e13942cb-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.074698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.074687    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.076939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.076918    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-kube-api-access-pldgj" pod "52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" (UID: "52eee9f7-0464-415f-ad3d-fbbdc4de3c5e"). InnerVolumeSpecName "kube-api-access-pldgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:23.086271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.086186    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"]
Oct 31 02:00:23.158563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.158516    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"]
Oct 31 02:00:23.177638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.177595    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pldgj\" (UniqueName: \"kubernetes.io/projected/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e-kube-api-access-pldgj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.272633 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.272602    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:23.495201 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.495162    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 02:00:23.495400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.495224    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 02:00:23.496205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.496182    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-kube-api-access-fnz4t\" (UniqueName: \"kubernetes.io/projected/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-kube-api-access-fnz4t\") pod \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\" (UID: \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\") "
Oct 31 02:00:23.496700 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.496668    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-maas-api-service-ca\") pod \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\" (UID: \"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660\") "
Oct 31 02:00:23.497136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.497006    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-maas-api-service-ca" pod "ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" (UID: "ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:23.512713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.512414    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.558697 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.558632    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-kube-api-access-fnz4t" pod "ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" (UID: "ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660"). InnerVolumeSpecName "kube-api-access-fnz4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:23.648554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.648521    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnz4t\" (UniqueName: \"kubernetes.io/projected/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660-kube-api-access-fnz4t\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.655692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.655666    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:23.749109 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.749069    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-kube-api-access-252vv\" (UniqueName: \"kubernetes.io/projected/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-kube-api-access-252vv\") pod \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\" (UID: \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\") "
Oct 31 02:00:23.749416 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.749391    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-maas-api-service-ca\") pod \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\" (UID: \"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6\") "
Oct 31 02:00:23.749731 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.749708    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-maas-api-service-ca" pod "3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" (UID: "3aa20e9f-a3fd-4e8b-82a0-dba557a665d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:23.750087 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.750024    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.751766 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.751739    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-kube-api-access-252vv" pod "3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" (UID: "3aa20e9f-a3fd-4e8b-82a0-dba557a665d6"). InnerVolumeSpecName "kube-api-access-252vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:23.851295 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.851213    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252vv\" (UniqueName: \"kubernetes.io/projected/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6-kube-api-access-252vv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.939240 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.939208    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9" event={"ID":"e5627e74-5b80-493a-bc75-2e42e13942cb","Type":"ContainerDied","Data":"313989a89c05d9492de7ed7b61425a7eda70ccdd5c8fd19f4b52cc5a6ca6bc96"}
Oct 31 02:00:23.939535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.939519    5293 scope.go:122] "RemoveContainer" containerID="b5d6cab64971944cade0cf4391beb24d68b6c4ce5fe7cd7594496ad03d22ab95"
Oct 31 02:00:23.939723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.939237    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"
Oct 31 02:00:23.941243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.941224    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"
Oct 31 02:00:23.941314 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.941267    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8" event={"ID":"a726d42d-1730-47aa-8a2d-1b453e4f4bab","Type":"ContainerDied","Data":"1c7de1016673b63b7d7b31d7c8e62a418c74a7eb438fa7f0f7a7d5b69d68eedd"}
Oct 31 02:00:23.948149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.948129    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"
Oct 31 02:00:23.948238 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.948193    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl" event={"ID":"ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660","Type":"ContainerDied","Data":"adc08c2dbf65582cd1424ed065f473b048d4e58260f4a2847cb75ba3cafe8073"}
Oct 31 02:00:23.959453 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.959432    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd" event={"ID":"3aa20e9f-a3fd-4e8b-82a0-dba557a665d6","Type":"ContainerDied","Data":"b292f290e44a49df38b1bd43b4842453e61651f453c924cb888f990e238b0617"}
Oct 31 02:00:23.959651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.959440    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"
Oct 31 02:00:23.960682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.960662    5293 scope.go:122] "RemoveContainer" containerID="2fefd0a83e6243fd3d28aedafe031eff84ecda87e31319b33baa7e92bca8bf8e"
Oct 31 02:00:23.967832 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.967236    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt" event={"ID":"52eee9f7-0464-415f-ad3d-fbbdc4de3c5e","Type":"ContainerDied","Data":"b6cec7bf911f1ba2fe570fc1f2e9cfc1c43f49f91c655ee7f592edf022d98f28"}
Oct 31 02:00:23.967832 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.967317    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"
Oct 31 02:00:23.990728 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.990681    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"]
Oct 31 02:00:23.993319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:23.993283    5293 scope.go:122] "RemoveContainer" containerID="a5c1708bcc9e4a30289f53deb518d6d24899a3db1fa758c88634eed9d3ff93e0"
Oct 31 02:00:24.393958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.392695    5293 scope.go:122] "RemoveContainer" containerID="aa404ec3c4350b1ecd5ca6bdb53c794ddef1114efda9f15c8574937326780576"
Oct 31 02:00:24.431220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.431173    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" path="/var/lib/kubelet/pods/aecd11ba-cc8a-48bb-8641-a0089eadbc07/volumes"
Oct 31 02:00:24.432168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.432146    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"]
Oct 31 02:00:24.442486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.442463    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890200-gc9k9"]
Oct 31 02:00:24.457561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.457534    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"]
Oct 31 02:00:24.458991 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.458958    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 02:00:24.459216 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.459013    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 02:00:24.475116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.471241    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890200-w2tvl"]
Oct 31 02:00:24.481117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.481093    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"]
Oct 31 02:00:24.486378 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.486358    5293 scope.go:122] "RemoveContainer" containerID="0ddafe16ba7cd7cd121500469f71f59a879807ab3f2644c0028d4844ec338856"
Oct 31 02:00:24.487138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.487116    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890200-xsktd"]
Oct 31 02:00:24.558246 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.558223    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"]
Oct 31 02:00:24.567515 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.566534    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890200-2zzz8"]
Oct 31 02:00:24.571853 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.571563    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:24.572400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.572384    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"]
Oct 31 02:00:24.576912 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.576821    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-6c54a1-29890200-fhlkt"]
Oct 31 02:00:24.632463 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.632431    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3681d17a-a7c0-45ac-9888-d44c117fc742-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3681d17a-a7c0-45ac-9888-d44c117fc742-maas-api-service-ca\") pod \"3681d17a-a7c0-45ac-9888-d44c117fc742\" (UID: \"3681d17a-a7c0-45ac-9888-d44c117fc742\") "
Oct 31 02:00:24.632630 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.632520    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3681d17a-a7c0-45ac-9888-d44c117fc742-kube-api-access-vcx7r\" (UniqueName: \"kubernetes.io/projected/3681d17a-a7c0-45ac-9888-d44c117fc742-kube-api-access-vcx7r\") pod \"3681d17a-a7c0-45ac-9888-d44c117fc742\" (UID: \"3681d17a-a7c0-45ac-9888-d44c117fc742\") "
Oct 31 02:00:24.633117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.633092    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3681d17a-a7c0-45ac-9888-d44c117fc742-maas-api-service-ca" pod "3681d17a-a7c0-45ac-9888-d44c117fc742" (UID: "3681d17a-a7c0-45ac-9888-d44c117fc742"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:24.644797 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.644741    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:24.658333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.655906    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3681d17a-a7c0-45ac-9888-d44c117fc742-kube-api-access-vcx7r" pod "3681d17a-a7c0-45ac-9888-d44c117fc742" (UID: "3681d17a-a7c0-45ac-9888-d44c117fc742"). InnerVolumeSpecName "kube-api-access-vcx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:24.726813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.726783    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:24.731235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.731207    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:24.733895 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.733867    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-maas-api-service-ca\") pod \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\" (UID: \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\") "
Oct 31 02:00:24.734033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.734016    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35bf319f-dfea-458f-be77-6c0823836fb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bf319f-dfea-458f-be77-6c0823836fb5-maas-api-service-ca\") pod \"35bf319f-dfea-458f-be77-6c0823836fb5\" (UID: \"35bf319f-dfea-458f-be77-6c0823836fb5\") "
Oct 31 02:00:24.734222 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.734205    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-kube-api-access-b6v5s\" (UniqueName: \"kubernetes.io/projected/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-kube-api-access-b6v5s\") pod \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\" (UID: \"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a\") "
Oct 31 02:00:24.734309 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.734284    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-maas-api-service-ca" pod "460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" (UID: "460f3b5d-e0cf-45d2-bbe6-a468cd04f24a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:24.734390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.734320    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bf319f-dfea-458f-be77-6c0823836fb5-kube-api-access-qtd44\" (UniqueName: \"kubernetes.io/projected/35bf319f-dfea-458f-be77-6c0823836fb5-kube-api-access-qtd44\") pod \"35bf319f-dfea-458f-be77-6c0823836fb5\" (UID: \"35bf319f-dfea-458f-be77-6c0823836fb5\") "
Oct 31 02:00:24.734390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.734354    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bf319f-dfea-458f-be77-6c0823836fb5-maas-api-service-ca" pod "35bf319f-dfea-458f-be77-6c0823836fb5" (UID: "35bf319f-dfea-458f-be77-6c0823836fb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:24.735133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.735113    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35bf319f-dfea-458f-be77-6c0823836fb5-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:24.735270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.735258    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3681d17a-a7c0-45ac-9888-d44c117fc742-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:24.735364 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.735349    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcx7r\" (UniqueName: \"kubernetes.io/projected/3681d17a-a7c0-45ac-9888-d44c117fc742-kube-api-access-vcx7r\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:24.735462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.735450    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:24.737347 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.737321    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bf319f-dfea-458f-be77-6c0823836fb5-kube-api-access-qtd44" pod "35bf319f-dfea-458f-be77-6c0823836fb5" (UID: "35bf319f-dfea-458f-be77-6c0823836fb5"). InnerVolumeSpecName "kube-api-access-qtd44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:24.737425 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.737367    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-kube-api-access-b6v5s" pod "460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" (UID: "460f3b5d-e0cf-45d2-bbe6-a468cd04f24a"). InnerVolumeSpecName "kube-api-access-b6v5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:24.838246 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.838210    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtd44\" (UniqueName: \"kubernetes.io/projected/35bf319f-dfea-458f-be77-6c0823836fb5-kube-api-access-qtd44\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:24.839907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.839884    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6v5s\" (UniqueName: \"kubernetes.io/projected/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a-kube-api-access-b6v5s\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:24.940913 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.940876    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227a89a1-09f8-4819-8526-1340b1e8ca1d-kube-api-access-g85vw\" (UniqueName: \"kubernetes.io/projected/227a89a1-09f8-4819-8526-1340b1e8ca1d-kube-api-access-g85vw\") pod \"227a89a1-09f8-4819-8526-1340b1e8ca1d\" (UID: \"227a89a1-09f8-4819-8526-1340b1e8ca1d\") "
Oct 31 02:00:24.941147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.940983    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227a89a1-09f8-4819-8526-1340b1e8ca1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/227a89a1-09f8-4819-8526-1340b1e8ca1d-maas-api-service-ca\") pod \"227a89a1-09f8-4819-8526-1340b1e8ca1d\" (UID: \"227a89a1-09f8-4819-8526-1340b1e8ca1d\") "
Oct 31 02:00:24.942185 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.942159    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227a89a1-09f8-4819-8526-1340b1e8ca1d-maas-api-service-ca" pod "227a89a1-09f8-4819-8526-1340b1e8ca1d" (UID: "227a89a1-09f8-4819-8526-1340b1e8ca1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:24.943426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.943403    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/227a89a1-09f8-4819-8526-1340b1e8ca1d-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:24.944341 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.944317    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227a89a1-09f8-4819-8526-1340b1e8ca1d-kube-api-access-g85vw" pod "227a89a1-09f8-4819-8526-1340b1e8ca1d" (UID: "227a89a1-09f8-4819-8526-1340b1e8ca1d"). InnerVolumeSpecName "kube-api-access-g85vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:24.979870 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.979834    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd" event={"ID":"3681d17a-a7c0-45ac-9888-d44c117fc742","Type":"ContainerDied","Data":"60bb6eacd8a333d8633c1b65d6407e09606a6c6edd55b5e471d761efee47ee7b"}
Oct 31 02:00:24.980055 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.979894    5293 scope.go:122] "RemoveContainer" containerID="d4f275d5d13088271f15e5341cbf848f03ad5b9038e921027c63793142d39c80"
Oct 31 02:00:24.980055 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.979966    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"
Oct 31 02:00:24.984256 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.984223    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l" event={"ID":"227a89a1-09f8-4819-8526-1340b1e8ca1d","Type":"ContainerDied","Data":"526c33758ae88d82cbe4e806ff5641bb4b6acaf97efab7db064807079778df29"}
Oct 31 02:00:24.984381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.984363    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"
Oct 31 02:00:24.992878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.991077    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"
Oct 31 02:00:24.992878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:24.991121    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb" event={"ID":"35bf319f-dfea-458f-be77-6c0823836fb5","Type":"ContainerDied","Data":"84107401bae0abef5add45c4add43cdd03816518e70fec5dde2ac36868bc696d"}
Oct 31 02:00:25.005568 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.005540    5293 scope.go:122] "RemoveContainer" containerID="a578019f13cc691e4b24bf30214f6086e82dd97409f1d29104ed107aef50648c"
Oct 31 02:00:25.061260 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.058924    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g85vw\" (UniqueName: \"kubernetes.io/projected/227a89a1-09f8-4819-8526-1340b1e8ca1d-kube-api-access-g85vw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:25.110919 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.106782    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk" event={"ID":"460f3b5d-e0cf-45d2-bbe6-a468cd04f24a","Type":"ContainerDied","Data":"168439cce9149334c545b928c1d2530f2b94ae8e6acdab6f776b9eea336b4e70"}
Oct 31 02:00:25.123064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.111236    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"]
Oct 31 02:00:25.123323 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.114878    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"
Oct 31 02:00:25.176060 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.159752    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890200-9c4dd"]
Oct 31 02:00:25.176060 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.172999    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"]
Oct 31 02:00:25.186696 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.184216    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-5ptbb"]
Oct 31 02:00:25.193703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.193635    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"]
Oct 31 02:00:25.198016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.197985    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890200-szk6l"]
Oct 31 02:00:25.200662 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.200637    5293 scope.go:122] "RemoveContainer" containerID="0b862fc9e63e08b35483c3085fdfbe621edec233d8742b1538a63b3609712a6e"
Oct 31 02:00:25.207757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.207730    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"]
Oct 31 02:00:25.211730 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.211708    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890200-89djk"]
Oct 31 02:00:25.218792 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.218750    5293 scope.go:122] "RemoveContainer" containerID="e0949f60c5957326414f107a066b13fb3e31ca0c1cb6a8d9b16207cf1e309d8e"
Oct 31 02:00:25.462681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.462613    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:25.463086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.463072    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:25.463882 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.463863    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:00:25.780345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.780273    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"]
Oct 31 02:00:25.781030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781016    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:25.781096 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781031    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:25.781096 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781071    5293 cpu_manager.go:414] "containerMap: removing container" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:25.781096 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781078    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:25.781096 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781092    5293 cpu_manager.go:414] "containerMap: removing container" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781097    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781106    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781111    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781119    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781124    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781134    5293 cpu_manager.go:414] "containerMap: removing container" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781139    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781146    5293 cpu_manager.go:414] "containerMap: removing container" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781150    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781156    5293 cpu_manager.go:414] "containerMap: removing container" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781161    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781167    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781172    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781180    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781184    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781191    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781196    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781202    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781206    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781214    5293 cpu_manager.go:414] "containerMap: removing container" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781218    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781228    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781232    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781240    5293 cpu_manager.go:414] "containerMap: removing container" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:25.781234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781244    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781249    5293 cpu_manager.go:414] "containerMap: removing container" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781254    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781263    5293 cpu_manager.go:414] "containerMap: removing container" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781267    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781274    5293 cpu_manager.go:414] "containerMap: removing container" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781278    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781288    5293 cpu_manager.go:414] "containerMap: removing container" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781292    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781297    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9f05d658-ef99-4344-b7a0-1d691c708cca" containerName="revoke-keys"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781302    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f05d658-ef99-4344-b7a0-1d691c708cca" containerName="revoke-keys"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781310    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781314    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781320    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781324    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781332    5293 cpu_manager.go:414] "containerMap: removing container" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781336    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781342    5293 cpu_manager.go:414] "containerMap: removing container" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781346    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781352    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781356    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781363    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781368    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781474    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781481    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781487    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781493    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781498    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781503    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781509    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781514    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f05d658-ef99-4344-b7a0-1d691c708cca" containerName="revoke-keys"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781518    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781523    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781529    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781537    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781543    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781547    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781552    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781557    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781566    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781573    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781578    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781584    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781590    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781596    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781601    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:25.781940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.781607    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" containerName="cleanup"
Oct 31 02:00:25.787387 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.787366    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.794953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.794924    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:00:25.795163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.794995    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:00:25.795646 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.795621    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"]
Oct 31 02:00:25.801278 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.801258    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:00:25.801366 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.801331    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:00:25.874810 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.874774    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc05133-0802-40d1-a375-5c05293962df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.874962 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.874842    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phj8h\" (UniqueName: \"kubernetes.io/projected/0cc05133-0802-40d1-a375-5c05293962df-kube-api-access-phj8h\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.874962 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.874940    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc05133-0802-40d1-a375-5c05293962df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.976202 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.976170    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc05133-0802-40d1-a375-5c05293962df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.976202 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.976203    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phj8h\" (UniqueName: \"kubernetes.io/projected/0cc05133-0802-40d1-a375-5c05293962df-kube-api-access-phj8h\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.976419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.976262    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc05133-0802-40d1-a375-5c05293962df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.976872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.976850    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc05133-0802-40d1-a375-5c05293962df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.978953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.978929    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc05133-0802-40d1-a375-5c05293962df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:25.989117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:25.989089    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phj8h\" (UniqueName: \"kubernetes.io/projected/0cc05133-0802-40d1-a375-5c05293962df-kube-api-access-phj8h\") pod \"maas-api-revoke-sub-aa729f0f-k7hcc\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:26.112995 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.112920    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:26.254852 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.254815    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" path="/var/lib/kubelet/pods/227a89a1-09f8-4819-8526-1340b1e8ca1d/volumes"
Oct 31 02:00:26.255896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.255870    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" path="/var/lib/kubelet/pods/35bf319f-dfea-458f-be77-6c0823836fb5/volumes"
Oct 31 02:00:26.256645 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.256625    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" path="/var/lib/kubelet/pods/3681d17a-a7c0-45ac-9888-d44c117fc742/volumes"
Oct 31 02:00:26.257769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.257749    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa20e9f-a3fd-4e8b-82a0-dba557a665d6" path="/var/lib/kubelet/pods/3aa20e9f-a3fd-4e8b-82a0-dba557a665d6/volumes"
Oct 31 02:00:26.258569 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.258497    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" path="/var/lib/kubelet/pods/460f3b5d-e0cf-45d2-bbe6-a468cd04f24a/volumes"
Oct 31 02:00:26.259278 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.259260    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52eee9f7-0464-415f-ad3d-fbbdc4de3c5e" path="/var/lib/kubelet/pods/52eee9f7-0464-415f-ad3d-fbbdc4de3c5e/volumes"
Oct 31 02:00:26.260380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.260362    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" path="/var/lib/kubelet/pods/a726d42d-1730-47aa-8a2d-1b453e4f4bab/volumes"
Oct 31 02:00:26.261115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.261022    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660" path="/var/lib/kubelet/pods/ce53e59e-62c5-4c1b-a5b2-2ff5a1c9a660/volumes"
Oct 31 02:00:26.265380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.265359    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" path="/var/lib/kubelet/pods/e5627e74-5b80-493a-bc75-2e42e13942cb/volumes"
Oct 31 02:00:26.518560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.518529    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"]
Oct 31 02:00:26.520344 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:00:26.520314    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc05133_0802_40d1_a375_5c05293962df.slice/crio-9debc9e083ab01a5d4c190128d1d2125e467b9c75cbf2dd85fecb51fdd82c9dc WatchSource:0}: Error finding container 9debc9e083ab01a5d4c190128d1d2125e467b9c75cbf2dd85fecb51fdd82c9dc: Status 404 returned error can't find the container with id 9debc9e083ab01a5d4c190128d1d2125e467b9c75cbf2dd85fecb51fdd82c9dc
Oct 31 02:00:26.907364 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.907332    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"]
Oct 31 02:00:26.908301 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908279    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:26.908301 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908300    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:26.908476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908318    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:26.908476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908323    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:26.908476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908329    5293 cpu_manager.go:414] "containerMap: removing container" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:26.908476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908335    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:26.908476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908352    5293 cpu_manager.go:414] "containerMap: removing container" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:26.908476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908356    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:26.908476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908363    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:26.908476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908368    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:26.908838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908483    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5627e74-5b80-493a-bc75-2e42e13942cb" containerName="cleanup"
Oct 31 02:00:26.908838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908491    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bf319f-dfea-458f-be77-6c0823836fb5" containerName="cleanup"
Oct 31 02:00:26.908838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908507    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="227a89a1-09f8-4819-8526-1340b1e8ca1d" containerName="cleanup"
Oct 31 02:00:26.908838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908513    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecd11ba-cc8a-48bb-8641-a0089eadbc07" containerName="cleanup"
Oct 31 02:00:26.908838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908520    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="460f3b5d-e0cf-45d2-bbe6-a468cd04f24a" containerName="cleanup"
Oct 31 02:00:26.908838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908527    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a726d42d-1730-47aa-8a2d-1b453e4f4bab" containerName="cleanup"
Oct 31 02:00:26.908838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.908532    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3681d17a-a7c0-45ac-9888-d44c117fc742" containerName="cleanup"
Oct 31 02:00:26.912611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.912585    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:26.916454 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.916428    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-bb5c51-dockercfg-n7j7f\""
Oct 31 02:00:26.917214 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.917176    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-bb5c51\""
Oct 31 02:00:26.925125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.925100    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"]
Oct 31 02:00:26.952034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.951999    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"]
Oct 31 02:00:26.956543 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.956523    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:26.967083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:26.967061    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"]
Oct 31 02:00:27.034011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.033979    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8b2c\" (UniqueName: \"kubernetes.io/projected/eb03a89a-bd26-436c-9669-9a289b79dad1-kube-api-access-v8b2c\") pod \"payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6\" (UID: \"eb03a89a-bd26-436c-9669-9a289b79dad1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:27.034266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.034082    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrzg\" (UniqueName: \"kubernetes.io/projected/3419e3d8-bec8-4eba-9e85-dd542ed634dc-kube-api-access-zlrzg\") pod \"payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5\" (UID: \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:27.034266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.034188    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb03a89a-bd26-436c-9669-9a289b79dad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6\" (UID: \"eb03a89a-bd26-436c-9669-9a289b79dad1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:27.034266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.034235    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3419e3d8-bec8-4eba-9e85-dd542ed634dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5\" (UID: \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:27.147065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.145074    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8b2c\" (UniqueName: \"kubernetes.io/projected/eb03a89a-bd26-436c-9669-9a289b79dad1-kube-api-access-v8b2c\") pod \"payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6\" (UID: \"eb03a89a-bd26-436c-9669-9a289b79dad1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:27.147065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.145201    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrzg\" (UniqueName: \"kubernetes.io/projected/3419e3d8-bec8-4eba-9e85-dd542ed634dc-kube-api-access-zlrzg\") pod \"payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5\" (UID: \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:27.147065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.145287    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb03a89a-bd26-436c-9669-9a289b79dad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6\" (UID: \"eb03a89a-bd26-436c-9669-9a289b79dad1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:27.147065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.145353    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3419e3d8-bec8-4eba-9e85-dd542ed634dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5\" (UID: \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:27.147065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.146597    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3419e3d8-bec8-4eba-9e85-dd542ed634dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5\" (UID: \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:27.149028 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.147907    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb03a89a-bd26-436c-9669-9a289b79dad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6\" (UID: \"eb03a89a-bd26-436c-9669-9a289b79dad1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:27.212110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.209787    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrzg\" (UniqueName: \"kubernetes.io/projected/3419e3d8-bec8-4eba-9e85-dd542ed634dc-kube-api-access-zlrzg\") pod \"payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5\" (UID: \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:27.212110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.210529    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8b2c\" (UniqueName: \"kubernetes.io/projected/eb03a89a-bd26-436c-9669-9a289b79dad1-kube-api-access-v8b2c\") pod \"payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6\" (UID: \"eb03a89a-bd26-436c-9669-9a289b79dad1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:27.281068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.275227    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:27.288074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.285559    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:27.344076 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.337686    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc" event={"ID":"0cc05133-0802-40d1-a375-5c05293962df","Type":"ContainerStarted","Data":"f1b223c35a9fa44a310d94c9641c0664f8d4028055d5aa1c214343908cad0b75"}
Oct 31 02:00:27.344076 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.337752    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc" event={"ID":"0cc05133-0802-40d1-a375-5c05293962df","Type":"ContainerStarted","Data":"9debc9e083ab01a5d4c190128d1d2125e467b9c75cbf2dd85fecb51fdd82c9dc"}
Oct 31 02:00:27.478093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.477956    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc" podStartSLOduration=2.477937554 podStartE2EDuration="2.477937554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:27.472612519 +0000 UTC m=+2385.421260910" watchObservedRunningTime="2026-10-31 02:00:27.477937554 +0000 UTC m=+2385.426585976"
Oct 31 02:00:27.933515 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.933107    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"]
Oct 31 02:00:27.938466 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:27.938438    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"]
Oct 31 02:00:28.339230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.339193    5293 generic.go:356] "Generic (PLEG): container finished" podID="0cc05133-0802-40d1-a375-5c05293962df" containerID="f1b223c35a9fa44a310d94c9641c0664f8d4028055d5aa1c214343908cad0b75" exitCode=0
Oct 31 02:00:28.339447 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.339318    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc" event={"ID":"0cc05133-0802-40d1-a375-5c05293962df","Type":"ContainerDied","Data":"f1b223c35a9fa44a310d94c9641c0664f8d4028055d5aa1c214343908cad0b75"}
Oct 31 02:00:28.341256 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.341172    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6" event={"ID":"eb03a89a-bd26-436c-9669-9a289b79dad1","Type":"ContainerStarted","Data":"8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d"}
Oct 31 02:00:28.341256 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.341220    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6" event={"ID":"eb03a89a-bd26-436c-9669-9a289b79dad1","Type":"ContainerStarted","Data":"0a94d32d0e3ccf05db84e9214904571ca92448d3a93ee7866d3acbf10645cc2e"}
Oct 31 02:00:28.341439 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.341289    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:28.344098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.344071    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5" event={"ID":"3419e3d8-bec8-4eba-9e85-dd542ed634dc","Type":"ContainerStarted","Data":"5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11"}
Oct 31 02:00:28.344241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.344121    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5" event={"ID":"3419e3d8-bec8-4eba-9e85-dd542ed634dc","Type":"ContainerStarted","Data":"ce9447b900a7371b9f219c4dcfd44e5dece04bd13cbb98ffd0968bbb3757c7b4"}
Oct 31 02:00:28.344241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.344181    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:28.376941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.376888    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6" podStartSLOduration=2.376872245 podStartE2EDuration="2.376872245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:28.374905902 +0000 UTC m=+2386.323554294" watchObservedRunningTime="2026-10-31 02:00:28.376872245 +0000 UTC m=+2386.325520638"
Oct 31 02:00:28.393779 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:28.393732    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5" podStartSLOduration=2.393720701 podStartE2EDuration="2.393720701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:28.391749756 +0000 UTC m=+2386.340398148" watchObservedRunningTime="2026-10-31 02:00:28.393720701 +0000 UTC m=+2386.342369092"
Oct 31 02:00:29.714309 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.714242    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:29.772080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.767727    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc05133-0802-40d1-a375-5c05293962df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc05133-0802-40d1-a375-5c05293962df-maas-api-xks-ca\") pod \"0cc05133-0802-40d1-a375-5c05293962df\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") "
Oct 31 02:00:29.772080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.768107    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc05133-0802-40d1-a375-5c05293962df-kube-api-access-phj8h\" (UniqueName: \"kubernetes.io/projected/0cc05133-0802-40d1-a375-5c05293962df-kube-api-access-phj8h\") pod \"0cc05133-0802-40d1-a375-5c05293962df\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") "
Oct 31 02:00:29.772080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.768215    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc05133-0802-40d1-a375-5c05293962df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc05133-0802-40d1-a375-5c05293962df-maas-api-openshift-ca\") pod \"0cc05133-0802-40d1-a375-5c05293962df\" (UID: \"0cc05133-0802-40d1-a375-5c05293962df\") "
Oct 31 02:00:29.795065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.781490    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc05133-0802-40d1-a375-5c05293962df-maas-api-openshift-ca" pod "0cc05133-0802-40d1-a375-5c05293962df" (UID: "0cc05133-0802-40d1-a375-5c05293962df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:29.820191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.798600    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc05133-0802-40d1-a375-5c05293962df-kube-api-access-phj8h" pod "0cc05133-0802-40d1-a375-5c05293962df" (UID: "0cc05133-0802-40d1-a375-5c05293962df"). InnerVolumeSpecName "kube-api-access-phj8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:29.820191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.813794    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phj8h\" (UniqueName: \"kubernetes.io/projected/0cc05133-0802-40d1-a375-5c05293962df-kube-api-access-phj8h\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:29.820191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.813823    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc05133-0802-40d1-a375-5c05293962df-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:29.839073 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.838299    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc05133-0802-40d1-a375-5c05293962df-maas-api-xks-ca" pod "0cc05133-0802-40d1-a375-5c05293962df" (UID: "0cc05133-0802-40d1-a375-5c05293962df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:00:29.926062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:29.924192    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc05133-0802-40d1-a375-5c05293962df-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:30.451695 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:30.451669    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"
Oct 31 02:00:30.451865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:30.451693    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc" event={"ID":"0cc05133-0802-40d1-a375-5c05293962df","Type":"ContainerDied","Data":"9debc9e083ab01a5d4c190128d1d2125e467b9c75cbf2dd85fecb51fdd82c9dc"}
Oct 31 02:00:30.451865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:30.451776    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9debc9e083ab01a5d4c190128d1d2125e467b9c75cbf2dd85fecb51fdd82c9dc"
Oct 31 02:00:39.494065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:39.491787    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:00:39.502077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:39.497573    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:00:41.780146 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:00:41.780095    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:00:46.286271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:46.286228    5293 scope.go:122] "RemoveContainer" containerID="a89bc306aa776c79cf37b3ef942cc7fdffe58921b9cea4c91cc12fb4d7b781a7"
Oct 31 02:00:46.297538 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:46.297512    5293 scope.go:122] "RemoveContainer" containerID="0f670f75c85109d48f743bbd85ac4e4f9875b4b4a50d133b4d03f45f1d34960f"
Oct 31 02:00:46.308328 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:46.308302    5293 scope.go:122] "RemoveContainer" containerID="e43b845a5f1663ed2a44395aa3fdd0cff1e2f5cf130a153eb549ba3ba6695bb1"
Oct 31 02:00:46.319704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:46.319680    5293 scope.go:122] "RemoveContainer" containerID="0460fd6dffa3dce30168355a9b02afc8d73078d50486c425c6c6c0052a292643"
Oct 31 02:00:46.330764 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:46.330745    5293 scope.go:122] "RemoveContainer" containerID="668d5c07b2b13073bf23ee594701e27d6115b82321c5260b0e2673f879cad185"
Oct 31 02:00:46.342817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:46.342795    5293 scope.go:122] "RemoveContainer" containerID="106867af56182e7d11888b128e5f30389d2d25f9263bd6877963dbb1d31f0a37"
Oct 31 02:00:50.357647 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.357615    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"]
Oct 31 02:00:50.358745 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.358727    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0cc05133-0802-40d1-a375-5c05293962df" containerName="revoke-keys"
Oct 31 02:00:50.358808 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.358749    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc05133-0802-40d1-a375-5c05293962df" containerName="revoke-keys"
Oct 31 02:00:50.358944 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.358933    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc05133-0802-40d1-a375-5c05293962df" containerName="revoke-keys"
Oct 31 02:00:50.365939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.365917    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:50.369847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.369819    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2186c3\""
Oct 31 02:00:50.369959 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.369910    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2186c3-dockercfg-t4vjw\""
Oct 31 02:00:50.373264 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.373242    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"]
Oct 31 02:00:50.396062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.396018    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"]
Oct 31 02:00:50.401818 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.401797    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:50.686066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.680459    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"]
Oct 31 02:00:50.731063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.730855    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l\" (UID: \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:50.731271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.731134    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5x9r\" (UniqueName: \"kubernetes.io/projected/4e4f3ca0-7e72-417d-9750-3be06260bd17-kube-api-access-m5x9r\") pod \"payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7\" (UID: \"4e4f3ca0-7e72-417d-9750-3be06260bd17\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:50.762189 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.731378    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w74r\" (UniqueName: \"kubernetes.io/projected/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-kube-api-access-7w74r\") pod \"payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l\" (UID: \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:50.762189 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.731427    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f3ca0-7e72-417d-9750-3be06260bd17-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7\" (UID: \"4e4f3ca0-7e72-417d-9750-3be06260bd17\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:50.872120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.872076    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l\" (UID: \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:50.872332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.872283    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5x9r\" (UniqueName: \"kubernetes.io/projected/4e4f3ca0-7e72-417d-9750-3be06260bd17-kube-api-access-m5x9r\") pod \"payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7\" (UID: \"4e4f3ca0-7e72-417d-9750-3be06260bd17\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:50.872478 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.872461    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w74r\" (UniqueName: \"kubernetes.io/projected/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-kube-api-access-7w74r\") pod \"payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l\" (UID: \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:50.872544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.872508    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f3ca0-7e72-417d-9750-3be06260bd17-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7\" (UID: \"4e4f3ca0-7e72-417d-9750-3be06260bd17\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:50.879423 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.873506    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f3ca0-7e72-417d-9750-3be06260bd17-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7\" (UID: \"4e4f3ca0-7e72-417d-9750-3be06260bd17\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:50.880483 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.880457    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l\" (UID: \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:50.937673 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.937309    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w74r\" (UniqueName: \"kubernetes.io/projected/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-kube-api-access-7w74r\") pod \"payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l\" (UID: \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:50.937673 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.937349    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5x9r\" (UniqueName: \"kubernetes.io/projected/4e4f3ca0-7e72-417d-9750-3be06260bd17-kube-api-access-m5x9r\") pod \"payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7\" (UID: \"4e4f3ca0-7e72-417d-9750-3be06260bd17\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:50.995064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.992352    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:50.995894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:50.995865    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:51.776245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:51.776215    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"]
Oct 31 02:00:51.784079 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:00:51.778547    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4f3ca0_7e72_417d_9750_3be06260bd17.slice/crio-6a2e3c0d4d28936bb44b30cd041f783058cda123e3acd7b83924c9032b50182f WatchSource:0}: Error finding container 6a2e3c0d4d28936bb44b30cd041f783058cda123e3acd7b83924c9032b50182f: Status 404 returned error can't find the container with id 6a2e3c0d4d28936bb44b30cd041f783058cda123e3acd7b83924c9032b50182f
Oct 31 02:00:52.119740 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:52.119698    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"]
Oct 31 02:00:52.308466 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:52.308432    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" event={"ID":"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa","Type":"ContainerStarted","Data":"43ef3198c180bd25f10b6fe24a3835e7997634403367b8e7b5f1d0de68567cc6"}
Oct 31 02:00:52.311033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:52.310918    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7" event={"ID":"4e4f3ca0-7e72-417d-9750-3be06260bd17","Type":"ContainerStarted","Data":"6a2e3c0d4d28936bb44b30cd041f783058cda123e3acd7b83924c9032b50182f"}
Oct 31 02:00:53.475082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:53.474206    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7" event={"ID":"4e4f3ca0-7e72-417d-9750-3be06260bd17","Type":"ContainerStarted","Data":"3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96"}
Oct 31 02:00:53.476116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:53.476001    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:00:53.479235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:53.479211    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" event={"ID":"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa","Type":"ContainerStarted","Data":"c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663"}
Oct 31 02:00:53.480516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:53.480497    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:00:53.578220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:53.578154    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7" podStartSLOduration=3.578130836 podStartE2EDuration="3.578130836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:53.504830121 +0000 UTC m=+2411.453478513" watchObservedRunningTime="2026-10-31 02:00:53.578130836 +0000 UTC m=+2411.526779228"
Oct 31 02:00:53.602065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:53.597347    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" podStartSLOduration=3.59732535 podStartE2EDuration="3.59732535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:53.524564396 +0000 UTC m=+2411.473212789" watchObservedRunningTime="2026-10-31 02:00:53.59732535 +0000 UTC m=+2411.545973746"
Oct 31 02:00:54.636442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:00:54.636402    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:01:01.668085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.667645    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"]
Oct 31 02:01:01.680009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.679958    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"]
Oct 31 02:01:01.680185 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.680101    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:01.684005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.683973    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:01:01.684217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.684054    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:01:01.684217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.684137    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:01:01.684217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.684177    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:01:01.912157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.905781    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:01.912157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.905839    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pn76\" (UniqueName: \"kubernetes.io/projected/8cc9a141-ddb1-4b88-9561-d8b504634594-kube-api-access-2pn76\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:01.912157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:01.905901    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:02.069118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:02.050120    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:02.069118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:02.050177    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pn76\" (UniqueName: \"kubernetes.io/projected/8cc9a141-ddb1-4b88-9561-d8b504634594-kube-api-access-2pn76\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:02.069118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:02.050222    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:02.069118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:02.062899    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:02.084634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:02.079694    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pn76\" (UniqueName: \"kubernetes.io/projected/8cc9a141-ddb1-4b88-9561-d8b504634594-kube-api-access-2pn76\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:02.084634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:02.080250    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9165178-kzjh4\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:02.329325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:02.329229    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:03.084637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:03.068561    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"]
Oct 31 02:01:04.343942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:04.343812    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4" event={"ID":"8cc9a141-ddb1-4b88-9561-d8b504634594","Type":"ContainerStarted","Data":"fb936a8bf140fc7607fefa6e1b00ed8fa00afb9380e4f45b8db75d2a6fab75e8"}
Oct 31 02:01:05.363832 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:05.363789    5293 generic.go:356] "Generic (PLEG): container finished" podID="8cc9a141-ddb1-4b88-9561-d8b504634594" containerID="4a33ee37d618cbd7f0c5be4b8dcae5ab563da7df64a21acc009ff421d3db3411" exitCode=0
Oct 31 02:01:05.364325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:05.363991    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4" event={"ID":"8cc9a141-ddb1-4b88-9561-d8b504634594","Type":"ContainerDied","Data":"4a33ee37d618cbd7f0c5be4b8dcae5ab563da7df64a21acc009ff421d3db3411"}
Oct 31 02:01:05.499487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:05.499390    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:01:05.501841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:05.501814    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:01:06.841983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:06.841420    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:06.876686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:06.876649    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc9a141-ddb1-4b88-9561-d8b504634594-kube-api-access-2pn76\" (UniqueName: \"kubernetes.io/projected/8cc9a141-ddb1-4b88-9561-d8b504634594-kube-api-access-2pn76\") pod \"8cc9a141-ddb1-4b88-9561-d8b504634594\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") "
Oct 31 02:01:06.879690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:06.879652    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc9a141-ddb1-4b88-9561-d8b504634594-kube-api-access-2pn76" pod "8cc9a141-ddb1-4b88-9561-d8b504634594" (UID: "8cc9a141-ddb1-4b88-9561-d8b504634594"). InnerVolumeSpecName "kube-api-access-2pn76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:06.978589 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:06.978534    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-openshift-ca\") pod \"8cc9a141-ddb1-4b88-9561-d8b504634594\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") "
Oct 31 02:01:06.978800 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:06.978731    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-xks-ca\") pod \"8cc9a141-ddb1-4b88-9561-d8b504634594\" (UID: \"8cc9a141-ddb1-4b88-9561-d8b504634594\") "
Oct 31 02:01:06.979725 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:06.979691    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pn76\" (UniqueName: \"kubernetes.io/projected/8cc9a141-ddb1-4b88-9561-d8b504634594-kube-api-access-2pn76\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:06.981027 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:06.980995    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-openshift-ca" pod "8cc9a141-ddb1-4b88-9561-d8b504634594" (UID: "8cc9a141-ddb1-4b88-9561-d8b504634594"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:06.982654 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:06.982599    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-xks-ca" pod "8cc9a141-ddb1-4b88-9561-d8b504634594" (UID: "8cc9a141-ddb1-4b88-9561-d8b504634594"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:01:07.081679 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:07.081629    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:07.081679 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:07.081673    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc9a141-ddb1-4b88-9561-d8b504634594-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:07.310294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:07.310239    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"]
Oct 31 02:01:07.310565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:07.310534    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy" containerID="cri-o://473a0dbcfbecf5ab9f425302bea2efadc6c4478cbca7080ac0d71e341f09c041" gracePeriod=30
Oct 31 02:01:07.626111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:07.625031    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4" event={"ID":"8cc9a141-ddb1-4b88-9561-d8b504634594","Type":"ContainerDied","Data":"fb936a8bf140fc7607fefa6e1b00ed8fa00afb9380e4f45b8db75d2a6fab75e8"}
Oct 31 02:01:07.626111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:07.625172    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb936a8bf140fc7607fefa6e1b00ed8fa00afb9380e4f45b8db75d2a6fab75e8"
Oct 31 02:01:07.626111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:07.625249    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"
Oct 31 02:01:08.024414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.018253    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"]
Oct 31 02:01:08.024414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.021315    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8cc9a141-ddb1-4b88-9561-d8b504634594" containerName="revoke-keys"
Oct 31 02:01:08.024414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.021338    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc9a141-ddb1-4b88-9561-d8b504634594" containerName="revoke-keys"
Oct 31 02:01:08.024414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.021575    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc9a141-ddb1-4b88-9561-d8b504634594" containerName="revoke-keys"
Oct 31 02:01:08.091076 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.078275    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"]
Oct 31 02:01:08.222082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.078285    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:08.222082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.140830    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f67d3a9-5038-4167-8a95-4d23fcd86719-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct\" (UID: \"1f67d3a9-5038-4167-8a95-4d23fcd86719\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:08.222082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.182825    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5n5m\" (UniqueName: \"kubernetes.io/projected/1f67d3a9-5038-4167-8a95-4d23fcd86719-kube-api-access-r5n5m\") pod \"payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct\" (UID: \"1f67d3a9-5038-4167-8a95-4d23fcd86719\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:08.250240 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.223657    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-91b6b1\""
Oct 31 02:01:08.308136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.288837    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f67d3a9-5038-4167-8a95-4d23fcd86719-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct\" (UID: \"1f67d3a9-5038-4167-8a95-4d23fcd86719\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:08.308136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.288899    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5n5m\" (UniqueName: \"kubernetes.io/projected/1f67d3a9-5038-4167-8a95-4d23fcd86719-kube-api-access-r5n5m\") pod \"payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct\" (UID: \"1f67d3a9-5038-4167-8a95-4d23fcd86719\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:08.308136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.290396    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f67d3a9-5038-4167-8a95-4d23fcd86719-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct\" (UID: \"1f67d3a9-5038-4167-8a95-4d23fcd86719\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:08.373079 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.363629    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-91b6b1-dockercfg-vszh2\""
Oct 31 02:01:08.478069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.467891    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5n5m\" (UniqueName: \"kubernetes.io/projected/1f67d3a9-5038-4167-8a95-4d23fcd86719-kube-api-access-r5n5m\") pod \"payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct\" (UID: \"1f67d3a9-5038-4167-8a95-4d23fcd86719\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:08.485063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.478849    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:08.971207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:08.971113    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"]
Oct 31 02:01:08.975192 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:01:08.975164    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f67d3a9_5038_4167_8a95_4d23fcd86719.slice/crio-de1a4d530724af1fb3435d208242edd192663f6fc27c89382520f3c44ad03fc3 WatchSource:0}: Error finding container de1a4d530724af1fb3435d208242edd192663f6fc27c89382520f3c44ad03fc3: Status 404 returned error can't find the container with id de1a4d530724af1fb3435d208242edd192663f6fc27c89382520f3c44ad03fc3
Oct 31 02:01:09.691899 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:09.691862    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct" event={"ID":"1f67d3a9-5038-4167-8a95-4d23fcd86719","Type":"ContainerStarted","Data":"f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723"}
Oct 31 02:01:09.692446 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:09.691945    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct" event={"ID":"1f67d3a9-5038-4167-8a95-4d23fcd86719","Type":"ContainerStarted","Data":"de1a4d530724af1fb3435d208242edd192663f6fc27c89382520f3c44ad03fc3"}
Oct 31 02:01:09.692446 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:09.692100    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:09.715764 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:09.715703    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct" podStartSLOduration=2.715685335 podStartE2EDuration="2.715685335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:09.708890136 +0000 UTC m=+2427.657538528" watchObservedRunningTime="2026-10-31 02:01:09.715685335 +0000 UTC m=+2427.664333733"
Oct 31 02:01:10.494869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.494832    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"]
Oct 31 02:01:10.497010 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.495133    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" containerID="cri-o://63dbdad0030647a6d4c9ebece335649da9feac4ef9dd9151927fa92bf3b0953c" gracePeriod=30
Oct 31 02:01:10.703871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.703829    5293 generic.go:356] "Generic (PLEG): container finished" podID="e1321636-259d-41a9-9a08-99daca44e499" containerID="a3a75510fa2a9792efd80a1b67515a5581db9c956c9d0130ff70cbda5e8afcfc" exitCode=0
Oct 31 02:01:10.704409 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.703909    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" event={"ID":"e1321636-259d-41a9-9a08-99daca44e499","Type":"ContainerDied","Data":"a3a75510fa2a9792efd80a1b67515a5581db9c956c9d0130ff70cbda5e8afcfc"}
Oct 31 02:01:10.704744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.704722    5293 scope.go:122] "RemoveContainer" containerID="a3a75510fa2a9792efd80a1b67515a5581db9c956c9d0130ff70cbda5e8afcfc"
Oct 31 02:01:10.706984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.706909    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:10.707129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.707030    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:10.707129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.707093    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:10.780206 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.780152    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:01:10.780590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.780534    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:01:10.995234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:10.995189    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 02:01:11.749111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.749017    5293 generic.go:356] "Generic (PLEG): container finished" podID="1eaac94e-52de-410f-a032-7615ba6c9992" containerID="1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad" exitCode=0
Oct 31 02:01:11.750644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.750599    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" event={"ID":"1eaac94e-52de-410f-a032-7615ba6c9992","Type":"ContainerDied","Data":"1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad"}
Oct 31 02:01:11.751463 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.751441    5293 scope.go:122] "RemoveContainer" containerID="1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad"
Oct 31 02:01:11.775803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.775201    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" event={"ID":"e1321636-259d-41a9-9a08-99daca44e499","Type":"ContainerStarted","Data":"a50fec06a82ce909922661733c8c00dac1ced094df28ea4a51691c862045072e"}
Oct 31 02:01:11.775803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.775226    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:11.775803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.775315    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:11.775803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.775362    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:11.890332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.875561    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"]
Oct 31 02:01:11.892483 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:01:11.892451    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:01:11.926236 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.926205    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:11.962359 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.962306    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"]
Oct 31 02:01:11.968068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.968016    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:01:11.971034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.969661    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:01:11.971034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.970301    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:01:12.040132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:11.997679    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.050209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.050176    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744p2\" (UniqueName: \"kubernetes.io/projected/f460eb08-fec3-45ea-a6e1-4ee09babab4e-kube-api-access-744p2\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.050616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.050592    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.061326 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.061299    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 02:01:12.062204 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.062177    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 02:01:12.062285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.062237    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 02:01:12.070100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.069818    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:01:12.152250 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.152214    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744p2\" (UniqueName: \"kubernetes.io/projected/f460eb08-fec3-45ea-a6e1-4ee09babab4e-kube-api-access-744p2\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.153102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.153022    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.153276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.153248    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.154103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.154077    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.155958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.155939    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.164929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.164904    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-744p2\" (UniqueName: \"kubernetes.io/projected/f460eb08-fec3-45ea-a6e1-4ee09babab4e-kube-api-access-744p2\") pod \"maas-api-revoke-sub-4d7c4745-sgrxq\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.206217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.199912    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"]
Oct 31 02:01:12.206217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.202068    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy" containerID="cri-o://b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059" gracePeriod=30
Oct 31 02:01:12.223448 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.223408    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"]
Oct 31 02:01:12.223938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.223888    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy" containerID="cri-o://5f07e899d95f8d4dbe4a2905433aff3fd337a83d78bc3f2d1d8de254e67f9f47" gracePeriod=30
Oct 31 02:01:12.376889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.376619    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:12.567606 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.566843    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:01:12.861069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.859291    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" event={"ID":"1eaac94e-52de-410f-a032-7615ba6c9992","Type":"ContainerStarted","Data":"4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6"}
Oct 31 02:01:12.867061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.864109    5293 generic.go:356] "Generic (PLEG): container finished" podID="dade4288-429f-4934-801f-19a34ad4eb14" containerID="473a0dbcfbecf5ab9f425302bea2efadc6c4478cbca7080ac0d71e341f09c041" exitCode=0
Oct 31 02:01:12.867061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.865301    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" event={"ID":"dade4288-429f-4934-801f-19a34ad4eb14","Type":"ContainerDied","Data":"473a0dbcfbecf5ab9f425302bea2efadc6c4478cbca7080ac0d71e341f09c041"}
Oct 31 02:01:12.958022 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:12.957983    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"]
Oct 31 02:01:13.063733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.063698    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 02:01:13.064074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.064003    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 02:01:13.236748 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.236272    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 02:01:13.333179 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.333140    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-certs\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.333845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.333816    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-socket\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.356763 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.334378    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-certs" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:13.356763 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.336340    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-socket" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:13.392061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.391917    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-data\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.392061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.391982    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-kube-api-access-zt6cc\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-kube-api-access-zt6cc\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.392061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.392031    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-istio-token\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-istio-token\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.392362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.392097    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-envoy\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.392362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.392162    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dade4288-429f-4934-801f-19a34ad4eb14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dade4288-429f-4934-801f-19a34ad4eb14-istiod-ca-cert\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.392362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.392207    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-credential-socket\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.392530 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.392384    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dade4288-429f-4934-801f-19a34ad4eb14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dade4288-429f-4934-801f-19a34ad4eb14-istio-podinfo\") pod \"dade4288-429f-4934-801f-19a34ad4eb14\" (UID: \"dade4288-429f-4934-801f-19a34ad4eb14\") "
Oct 31 02:01:13.428129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.410422    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.428129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.410466    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.461418 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.461377    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-data" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:13.489106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.488960    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-credential-socket" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:13.490353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.490164    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dade4288-429f-4934-801f-19a34ad4eb14-istiod-ca-cert" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:13.529343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.523923    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.529343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.523959    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dade4288-429f-4934-801f-19a34ad4eb14-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.529343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.523975    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.576400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.573483    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-envoy" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:13.576400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.574878    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-kube-api-access-zt6cc" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "kube-api-access-zt6cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:13.576400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.575118    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dade4288-429f-4934-801f-19a34ad4eb14-istio-podinfo" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:13.576400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.576322    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-istio-token" pod "dade4288-429f-4934-801f-19a34ad4eb14" (UID: "dade4288-429f-4934-801f-19a34ad4eb14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:13.659298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.659254    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dade4288-429f-4934-801f-19a34ad4eb14-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.659525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.659509    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt6cc\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-kube-api-access-zt6cc\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.659630 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.659617    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dade4288-429f-4934-801f-19a34ad4eb14-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.659724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.659713    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dade4288-429f-4934-801f-19a34ad4eb14-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:13.762170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.762143    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:01:13.763539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.763287    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 31 02:01:13.763539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.763335    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 31 02:01:13.966178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.966141    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq" event={"ID":"f460eb08-fec3-45ea-a6e1-4ee09babab4e","Type":"ContainerStarted","Data":"a20de8efad4ad0359d2b906c214e855814b11ad1a7397a106c7c2923521e0722"}
Oct 31 02:01:13.966807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.966761    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq" event={"ID":"f460eb08-fec3-45ea-a6e1-4ee09babab4e","Type":"ContainerStarted","Data":"9206301fb64667d61087687c74ad46f1578be45205d3b9c4b36ca28f567ea664"}
Oct 31 02:01:13.969282 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.969264    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"
Oct 31 02:01:13.969440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.969301    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw" event={"ID":"dade4288-429f-4934-801f-19a34ad4eb14","Type":"ContainerDied","Data":"42cce3deca1c43146d5f158663c746b0a77be25530a885cd119a18996b3d797a"}
Oct 31 02:01:13.969562 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:13.969552    5293 scope.go:122] "RemoveContainer" containerID="473a0dbcfbecf5ab9f425302bea2efadc6c4478cbca7080ac0d71e341f09c041"
Oct 31 02:01:14.157933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.087417    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq" podStartSLOduration=3.087395165 podStartE2EDuration="3.087395165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:14.028900427 +0000 UTC m=+2431.977548823" watchObservedRunningTime="2026-10-31 02:01:14.087395165 +0000 UTC m=+2432.036043557"
Oct 31 02:01:14.157933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.111564    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 02:01:14.157933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.111620    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 02:01:14.283668 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.283629    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"]
Oct 31 02:01:14.283872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.283680    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-qv5lw"]
Oct 31 02:01:14.773285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.767149    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 31 02:01:14.773285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.767275    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 31 02:01:14.988429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.988127    5293 generic.go:356] "Generic (PLEG): container finished" podID="f460eb08-fec3-45ea-a6e1-4ee09babab4e" containerID="a20de8efad4ad0359d2b906c214e855814b11ad1a7397a106c7c2923521e0722" exitCode=0
Oct 31 02:01:14.988429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:14.988198    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq" event={"ID":"f460eb08-fec3-45ea-a6e1-4ee09babab4e","Type":"ContainerDied","Data":"a20de8efad4ad0359d2b906c214e855814b11ad1a7397a106c7c2923521e0722"}
Oct 31 02:01:15.064523 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:15.064424    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 02:01:15.064523 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:15.064496    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 02:01:15.767066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:15.764528    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 31 02:01:15.767066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:15.764589    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 31 02:01:16.046075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.039200    5293 generic.go:356] "Generic (PLEG): container finished" podID="b09ad6fd-125c-4f54-81c5-5862579af346" containerID="63dbdad0030647a6d4c9ebece335649da9feac4ef9dd9151927fa92bf3b0953c" exitCode=0
Oct 31 02:01:16.046075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.039543    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" event={"ID":"b09ad6fd-125c-4f54-81c5-5862579af346","Type":"ContainerDied","Data":"63dbdad0030647a6d4c9ebece335649da9feac4ef9dd9151927fa92bf3b0953c"}
Oct 31 02:01:16.068068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.061672    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 02:01:16.068068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.061730    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 02:01:16.165099 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.164268    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 02:01:16.186072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.181606    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dade4288-429f-4934-801f-19a34ad4eb14" path="/var/lib/kubelet/pods/dade4288-429f-4934-801f-19a34ad4eb14/volumes"
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195225    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-socket\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195333    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b09ad6fd-125c-4f54-81c5-5862579af346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b09ad6fd-125c-4f54-81c5-5862579af346-istiod-ca-cert\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195422    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-credential-socket\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195509    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-data\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195545    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-certs\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195561    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-istio-token\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-istio-token\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195631    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-kube-api-access-th7qc\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-kube-api-access-th7qc\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195669    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b09ad6fd-125c-4f54-81c5-5862579af346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b09ad6fd-125c-4f54-81c5-5862579af346-istio-podinfo\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.212082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.195734    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-envoy\") pod \"b09ad6fd-125c-4f54-81c5-5862579af346\" (UID: \"b09ad6fd-125c-4f54-81c5-5862579af346\") "
Oct 31 02:01:16.227089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.217132    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-envoy" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.267454    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-credential-socket" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.278130    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-socket" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.321259    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09ad6fd-125c-4f54-81c5-5862579af346-istiod-ca-cert" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.343279    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-certs" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.374193    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.374224    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.374240    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b09ad6fd-125c-4f54-81c5-5862579af346-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.374260    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.374277    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.427091    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-kube-api-access-th7qc" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "kube-api-access-th7qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.427235    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-istio-token" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.438717    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b09ad6fd-125c-4f54-81c5-5862579af346-istio-podinfo" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:16.454773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.454395    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-data" pod "b09ad6fd-125c-4f54-81c5-5862579af346" (UID: "b09ad6fd-125c-4f54-81c5-5862579af346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:16.477252 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.477189    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b09ad6fd-125c-4f54-81c5-5862579af346-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.477252 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.477255    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.477493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.477273    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7qc\" (UniqueName: \"kubernetes.io/projected/b09ad6fd-125c-4f54-81c5-5862579af346-kube-api-access-th7qc\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.477493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.477312    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b09ad6fd-125c-4f54-81c5-5862579af346-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.579797 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.579762    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:16.680172 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.680134    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-xks-ca\") pod \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") "
Oct 31 02:01:16.680372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.680298    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f460eb08-fec3-45ea-a6e1-4ee09babab4e-kube-api-access-744p2\" (UniqueName: \"kubernetes.io/projected/f460eb08-fec3-45ea-a6e1-4ee09babab4e-kube-api-access-744p2\") pod \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") "
Oct 31 02:01:16.680458 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.680400    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-openshift-ca\") pod \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\" (UID: \"f460eb08-fec3-45ea-a6e1-4ee09babab4e\") "
Oct 31 02:01:16.681572 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.681541    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-openshift-ca" pod "f460eb08-fec3-45ea-a6e1-4ee09babab4e" (UID: "f460eb08-fec3-45ea-a6e1-4ee09babab4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:16.683709 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.683681    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f460eb08-fec3-45ea-a6e1-4ee09babab4e-kube-api-access-744p2" pod "f460eb08-fec3-45ea-a6e1-4ee09babab4e" (UID: "f460eb08-fec3-45ea-a6e1-4ee09babab4e"). InnerVolumeSpecName "kube-api-access-744p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:16.686945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.686917    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-xks-ca" pod "f460eb08-fec3-45ea-a6e1-4ee09babab4e" (UID: "f460eb08-fec3-45ea-a6e1-4ee09babab4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:01:16.768277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.767670    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 31 02:01:16.768277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.767730    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 31 02:01:16.788187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.787623    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-744p2\" (UniqueName: \"kubernetes.io/projected/f460eb08-fec3-45ea-a6e1-4ee09babab4e-kube-api-access-744p2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.788187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.787650    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:16.788187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:16.787660    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f460eb08-fec3-45ea-a6e1-4ee09babab4e-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.086972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.086873    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 02:01:17.087615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.087577    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 02:01:17.217066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.201621    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw" event={"ID":"b09ad6fd-125c-4f54-81c5-5862579af346","Type":"ContainerDied","Data":"612779b000360772e4c138baf23541ad8fbea0a2136ffe37453bcca3aa35a65d"}
Oct 31 02:01:17.217066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.201726    5293 scope.go:122] "RemoveContainer" containerID="63dbdad0030647a6d4c9ebece335649da9feac4ef9dd9151927fa92bf3b0953c"
Oct 31 02:01:17.217066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.201943    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"
Oct 31 02:01:17.217066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.212823    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq" event={"ID":"f460eb08-fec3-45ea-a6e1-4ee09babab4e","Type":"ContainerDied","Data":"9206301fb64667d61087687c74ad46f1578be45205d3b9c4b36ca28f567ea664"}
Oct 31 02:01:17.217066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.212912    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9206301fb64667d61087687c74ad46f1578be45205d3b9c4b36ca28f567ea664"
Oct 31 02:01:17.217066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.213695    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"
Oct 31 02:01:17.514178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.514100    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"]
Oct 31 02:01:17.570101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.569716    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dtwnw"]
Oct 31 02:01:17.690102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.689622    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"]
Oct 31 02:01:17.690102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.689824    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy" containerID="cri-o://6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561" gracePeriod=30
Oct 31 02:01:17.862065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.859014    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 31 02:01:17.862065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:17.859088    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 31 02:01:18.066741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.066418    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Oct 31 02:01:18.066741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.066485    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Oct 31 02:01:18.091522 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.090561    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 02:01:18.184816 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.184768    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" path="/var/lib/kubelet/pods/b09ad6fd-125c-4f54-81c5-5862579af346/volumes"
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.262690    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-credential-socket\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.262797    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-certs\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.262859    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-envoy\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.262996    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b701fb-9871-4274-a0d0-b11054989cd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b701fb-9871-4274-a0d0-b11054989cd9-istiod-ca-cert\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.264120    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-podinfo\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.264288    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-data\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.264328    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-token\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-token\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.264369    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-socket\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.267649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.264480    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-kube-api-access-6fcq7\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-kube-api-access-6fcq7\") pod \"b1b701fb-9871-4274-a0d0-b11054989cd9\" (UID: \"b1b701fb-9871-4274-a0d0-b11054989cd9\") "
Oct 31 02:01:18.278085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.275206    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-certs" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.278085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.275578    5293 generic.go:356] "Generic (PLEG): container finished" podID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerID="5f07e899d95f8d4dbe4a2905433aff3fd337a83d78bc3f2d1d8de254e67f9f47" exitCode=0
Oct 31 02:01:18.278085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.275755    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" event={"ID":"1eb546a5-2b16-4018-a783-07a8fd2bacf7","Type":"ContainerDied","Data":"5f07e899d95f8d4dbe4a2905433aff3fd337a83d78bc3f2d1d8de254e67f9f47"}
Oct 31 02:01:18.278085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.275818    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f" event={"ID":"1eb546a5-2b16-4018-a783-07a8fd2bacf7","Type":"ContainerDied","Data":"4a7ed61d460d94091b3c65e5c2ff0926fc2c8960097bbbd8c99c69d835ec131d"}
Oct 31 02:01:18.278085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.275844    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a7ed61d460d94091b3c65e5c2ff0926fc2c8960097bbbd8c99c69d835ec131d"
Oct 31 02:01:18.278085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.277691    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-credential-socket" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.278955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.278874    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-data" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.279432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.279404    5293 generic.go:356] "Generic (PLEG): container finished" podID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerID="b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059" exitCode=0
Oct 31 02:01:18.279578 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.279557    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" event={"ID":"b1b701fb-9871-4274-a0d0-b11054989cd9","Type":"ContainerDied","Data":"b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059"}
Oct 31 02:01:18.279653 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.279612    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl" event={"ID":"b1b701fb-9871-4274-a0d0-b11054989cd9","Type":"ContainerDied","Data":"c3298a83365f8bec18082f2ad4796193897939da6a657c62de0b4033e8497348"}
Oct 31 02:01:18.279653 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.279644    5293 scope.go:122] "RemoveContainer" containerID="b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059"
Oct 31 02:01:18.279781 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.279764    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"
Oct 31 02:01:18.281021 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.280993    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-envoy" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.281453 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.281429    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-podinfo" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:18.283228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.283198    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-kube-api-access-6fcq7" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "kube-api-access-6fcq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:18.285089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.284969    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-socket" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.285959 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.285936    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b701fb-9871-4274-a0d0-b11054989cd9-istiod-ca-cert" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:18.287625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.286491    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 02:01:18.287625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.287580    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-token" pod "b1b701fb-9871-4274-a0d0-b11054989cd9" (UID: "b1b701fb-9871-4274-a0d0-b11054989cd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:18.368537 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.368506    5293 scope.go:122] "RemoveContainer" containerID="b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059"
Oct 31 02:01:18.369282 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:01:18.369242    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059\": container with ID starting with b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059 not found: ID does not exist" containerID="b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059"
Oct 31 02:01:18.369501 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369473    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059"} err="failed to get container status \"b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059\": rpc error: code = NotFound desc = could not find container \"b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059\": container with ID starting with b199e1a18c3594bfa0fd38fd33974b4617fc693a6f76064fe11c9e1de75d9059 not found: ID does not exist"
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369833    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b701fb-9871-4274-a0d0-b11054989cd9-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369858    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369882    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369896    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369911    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369928    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fcq7\" (UniqueName: \"kubernetes.io/projected/b1b701fb-9871-4274-a0d0-b11054989cd9-kube-api-access-6fcq7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369943    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369957    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.370253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.369970    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b701fb-9871-4274-a0d0-b11054989cd9-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.471241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471142    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-data\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.471241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471221    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istiod-ca-cert\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471293    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-credential-socket\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471324    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-envoy\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471388    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-certs\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471422    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-kube-api-access-t68lk\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-kube-api-access-t68lk\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471529    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-podinfo\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471780    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-token\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.471853    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-socket\") pod \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\" (UID: \"1eb546a5-2b16-4018-a783-07a8fd2bacf7\") "
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.485406    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-data" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.485814    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istiod-ca-cert" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.486118    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-credential-socket" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.492293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.486503    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-socket" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.509667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.508030    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-certs" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.568424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.529285    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-envoy" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:18.573520 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.571446    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-token" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:18.575189 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.573780    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-podinfo" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:18.575901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.575765    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.575901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.575796    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.575901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.575812    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.575901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.575825    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.575901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.575839    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.575901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.575852    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.575901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.575865    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb546a5-2b16-4018-a783-07a8fd2bacf7-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.575901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.575880    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb546a5-2b16-4018-a783-07a8fd2bacf7-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.578167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.578087    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-kube-api-access-t68lk" pod "1eb546a5-2b16-4018-a783-07a8fd2bacf7" (UID: "1eb546a5-2b16-4018-a783-07a8fd2bacf7"). InnerVolumeSpecName "kube-api-access-t68lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:18.662847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.662815    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"]
Oct 31 02:01:18.676388 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.676357    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-77shl"]
Oct 31 02:01:18.677379 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.677356    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t68lk\" (UniqueName: \"kubernetes.io/projected/1eb546a5-2b16-4018-a783-07a8fd2bacf7-kube-api-access-t68lk\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:18.762009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.761971    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 31 02:01:18.762214 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:18.762033    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 31 02:01:19.066285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.066258    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 02:01:19.066830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.066803    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 02:01:19.067822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.067787    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 02:01:19.366945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.366866    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"
Oct 31 02:01:19.421158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.421117    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"]
Oct 31 02:01:19.428590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.428560    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-lnd5f"]
Oct 31 02:01:19.767211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.767179    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:01:19.773070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.769778    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:01:19.786869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:19.786834    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:01:20.176634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:20.176600    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" path="/var/lib/kubelet/pods/1eb546a5-2b16-4018-a783-07a8fd2bacf7/volumes"
Oct 31 02:01:20.177193 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:20.177177    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" path="/var/lib/kubelet/pods/b1b701fb-9871-4274-a0d0-b11054989cd9/volumes"
Oct 31 02:01:20.707945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:20.707917    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:01:23.376106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.376072    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 02:01:23.409161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.409127    5293 generic.go:356] "Generic (PLEG): container finished" podID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerID="6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561" exitCode=0
Oct 31 02:01:23.409481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.409191    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"
Oct 31 02:01:23.409626 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.409210    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" event={"ID":"0b5860d1-93c7-4f7b-8c19-f08c0197e06e","Type":"ContainerDied","Data":"6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561"}
Oct 31 02:01:23.409759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.409745    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t" event={"ID":"0b5860d1-93c7-4f7b-8c19-f08c0197e06e","Type":"ContainerDied","Data":"6f0764e209fe72c1783c60bde7a63f0d6903062cc9499c842b20ad1956615f30"}
Oct 31 02:01:23.409891 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.409867    5293 scope.go:122] "RemoveContainer" containerID="6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561"
Oct 31 02:01:23.426708 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.426662    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-socket\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.426855 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.426754    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istiod-ca-cert\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.426855 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.426816    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-envoy\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.427321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.426981    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-kube-api-access-tl6qj\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-kube-api-access-tl6qj\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.427321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.427082    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-credential-socket\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.427321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.427148    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-podinfo\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.427321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.427222    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-token\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-token\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.427610 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.427343    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-data\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.427610 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.427520    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-certs\") pod \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\" (UID: \"0b5860d1-93c7-4f7b-8c19-f08c0197e06e\") "
Oct 31 02:01:23.428759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.428730    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-certs" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:23.429005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.428983    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-socket" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:23.429823 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.429788    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istiod-ca-cert" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:23.433095 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.432922    5293 scope.go:122] "RemoveContainer" containerID="6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561"
Oct 31 02:01:23.434223 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:01:23.434139    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561\": container with ID starting with 6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561 not found: ID does not exist" containerID="6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561"
Oct 31 02:01:23.434223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.434176    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561"} err="failed to get container status \"6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561\": rpc error: code = NotFound desc = could not find container \"6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561\": container with ID starting with 6eac31162ced2764dac8b83fe180a65ec707371cf4e02cfc14955cbc6851b561 not found: ID does not exist"
Oct 31 02:01:23.434923 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.434829    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-data" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:23.434923 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.434897    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-credential-socket" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:23.435549 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.435525    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-envoy" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:23.437534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.437507    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-token" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:23.438311 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.438288    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-kube-api-access-tl6qj" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "kube-api-access-tl6qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:23.442273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.439734    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-podinfo" pod "0b5860d1-93c7-4f7b-8c19-f08c0197e06e" (UID: "0b5860d1-93c7-4f7b-8c19-f08c0197e06e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530644    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530681    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530695    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530708    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530722    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530738    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530751    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530765    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.532131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.530778    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6qj\" (UniqueName: \"kubernetes.io/projected/0b5860d1-93c7-4f7b-8c19-f08c0197e06e-kube-api-access-tl6qj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:23.788967 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.788934    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"]
Oct 31 02:01:23.789303 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:23.789283    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x947t"]
Oct 31 02:01:24.217064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:24.216733    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" path="/var/lib/kubelet/pods/0b5860d1-93c7-4f7b-8c19-f08c0197e06e/volumes"
Oct 31 02:01:28.455750 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:28.455714    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"]
Oct 31 02:01:29.142072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:29.092853    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"]
Oct 31 02:01:29.142072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:29.093181    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy" containerID="cri-o://421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35" gracePeriod=30
Oct 31 02:01:30.488611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.488583    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"]
Oct 31 02:01:30.489369 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489353    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f460eb08-fec3-45ea-a6e1-4ee09babab4e" containerName="revoke-keys"
Oct 31 02:01:30.489432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489371    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f460eb08-fec3-45ea-a6e1-4ee09babab4e" containerName="revoke-keys"
Oct 31 02:01:30.489432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489391    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy"
Oct 31 02:01:30.489432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489397    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy"
Oct 31 02:01:30.489432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489429    5293 cpu_manager.go:414] "containerMap: removing container" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy"
Oct 31 02:01:30.489666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489435    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy"
Oct 31 02:01:30.489666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489447    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy"
Oct 31 02:01:30.489666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489451    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy"
Oct 31 02:01:30.489666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489462    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy"
Oct 31 02:01:30.489666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489467    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy"
Oct 31 02:01:30.489666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489493    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy"
Oct 31 02:01:30.489666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489497    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy"
Oct 31 02:01:30.489905 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489667    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b701fb-9871-4274-a0d0-b11054989cd9" containerName="istio-proxy"
Oct 31 02:01:30.489905 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489693    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f460eb08-fec3-45ea-a6e1-4ee09babab4e" containerName="revoke-keys"
Oct 31 02:01:30.489905 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489708    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09ad6fd-125c-4f54-81c5-5862579af346" containerName="istio-proxy"
Oct 31 02:01:30.489905 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489718    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="dade4288-429f-4934-801f-19a34ad4eb14" containerName="istio-proxy"
Oct 31 02:01:30.489905 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489728    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb546a5-2b16-4018-a783-07a8fd2bacf7" containerName="istio-proxy"
Oct 31 02:01:30.489905 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.489740    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5860d1-93c7-4f7b-8c19-f08c0197e06e" containerName="istio-proxy"
Oct 31 02:01:30.494863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.494841    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.498786 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.498765    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-28e4f3-openshift-default-dockercfg-xf2nk\""
Oct 31 02:01:30.542114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.541733    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"]
Oct 31 02:01:30.618549 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.618511    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-envoy\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.618835 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.618815    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-data\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.618992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.618975    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-credential-socket\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.620151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.620127    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-istio-token\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.778122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.771837    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e177db-8776-47df-96ad-404ec445b4b6-istiod-ca-cert\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.778122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.772066    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e177db-8776-47df-96ad-404ec445b4b6-istio-podinfo\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.778122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.772192    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-socket\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.778122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.772228    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjlk\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-kube-api-access-fbjlk\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.778122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.772432    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-certs\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.877225 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.877183    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e177db-8776-47df-96ad-404ec445b4b6-istio-podinfo\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.877576 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.877553    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-socket\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.877707 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.877691    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjlk\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-kube-api-access-fbjlk\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.877981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.877941    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-certs\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.878347 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.878315    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-envoy\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.878509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.878490    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-data\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.878681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.878664    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-credential-socket\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.878890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.878873    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-istio-token\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.879021 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.879007    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e177db-8776-47df-96ad-404ec445b4b6-istiod-ca-cert\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.880012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.879988    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e177db-8776-47df-96ad-404ec445b4b6-istiod-ca-cert\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.881741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.881716    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-socket\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.884521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.882332    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-data\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.884649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.882517    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-certs\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.884649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.884458    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-credential-socket\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.888983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.888801    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-envoy\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.912270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.912238    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e177db-8776-47df-96ad-404ec445b4b6-istio-podinfo\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.933349 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.933318    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-istio-token\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:30.957112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:30.942745    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjlk\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-kube-api-access-fbjlk\") pod \"e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") " pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:31.113837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.113742    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:31.460595 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:01:31.460561    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e177db_8776_47df_96ad_404ec445b4b6.slice/crio-c5a68689af00f565544a252cf58972e0cb26baa17c18a12f250ec21a2daaac44 WatchSource:0}: Error finding container c5a68689af00f565544a252cf58972e0cb26baa17c18a12f250ec21a2daaac44: Status 404 returned error can't find the container with id c5a68689af00f565544a252cf58972e0cb26baa17c18a12f250ec21a2daaac44
Oct 31 02:01:31.460847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.460819    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"]
Oct 31 02:01:31.464259 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.464214    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:31.464372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.464308    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:31.464442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.464381    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:31.579054 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.576322    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" event={"ID":"e8e177db-8776-47df-96ad-404ec445b4b6","Type":"ContainerStarted","Data":"c5a68689af00f565544a252cf58972e0cb26baa17c18a12f250ec21a2daaac44"}
Oct 31 02:01:31.655814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.653657    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"]
Oct 31 02:01:31.659342 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.659313    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.664325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.664297    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-df9d42-openshift-default-dockercfg-thsmn\""
Oct 31 02:01:31.685782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.685751    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"]
Oct 31 02:01:31.818414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818322    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-data\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.818590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818424    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-token\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.818590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818568    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-podinfo\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.818699 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818629    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-socket\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.818699 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818657    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-certs\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.818699 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818681    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt2m5\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-kube-api-access-mt2m5\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.818865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818716    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istiod-ca-cert\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.818865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818750    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-envoy\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.818865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.818800    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-credential-socket\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922141    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-data\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922238    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-token\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922392    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-podinfo\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922457    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-socket\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922487    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-certs\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922513    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2m5\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-kube-api-access-mt2m5\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922552    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istiod-ca-cert\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922587    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-envoy\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.922639    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-credential-socket\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.923294    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-credential-socket\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.943084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.923544    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-data\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.945129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.944549    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istiod-ca-cert\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.945129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.944819    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-socket\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.946600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.945894    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-certs\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.984085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.973720    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2m5\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-kube-api-access-mt2m5\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:31.988820 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:31.988750    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-podinfo\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:32.009134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.004371    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-envoy\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:32.048617 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.048585    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-token\") pod \"e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") " pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:32.284798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.284762    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:32.639022 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.638934    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" event={"ID":"e8e177db-8776-47df-96ad-404ec445b4b6","Type":"ContainerStarted","Data":"5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99"}
Oct 31 02:01:32.641136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.640806    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"]
Oct 31 02:01:32.652435 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.652373    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:32.652665 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.652627    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:32.652824 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.652795    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:32.673081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:32.672880    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podStartSLOduration=2.67286065 podStartE2EDuration="2.67286065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:32.667291794 +0000 UTC m=+2450.615940188" watchObservedRunningTime="2026-10-31 02:01:32.67286065 +0000 UTC m=+2450.621509044"
Oct 31 02:01:33.114331 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:33.114296    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:33.121445 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:33.121416    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:33.657934 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:33.657889    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" event={"ID":"fc5aa93f-da9b-41e2-acc1-155a54a3767d","Type":"ContainerStarted","Data":"b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519"}
Oct 31 02:01:33.658440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:33.658006    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" event={"ID":"fc5aa93f-da9b-41e2-acc1-155a54a3767d","Type":"ContainerStarted","Data":"5f8e90837a6c4847dfdbf0e48762c4758a45cd61a611a3c51f41e6b677dfd265"}
Oct 31 02:01:33.659200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:33.659173    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:33.660613 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:33.660594    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:01:33.685545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:33.685501    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podStartSLOduration=2.685488286 podStartE2EDuration="2.685488286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:33.680341088 +0000 UTC m=+2451.628989479" watchObservedRunningTime="2026-10-31 02:01:33.685488286 +0000 UTC m=+2451.634136677"
Oct 31 02:01:34.285691 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.285665    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:34.298174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.298141    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:34.421865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.421809    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 02:01:34.476214 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476177    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-data\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.476405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476327    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99da1e81-f392-47a3-962a-cec0233e0288-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99da1e81-f392-47a3-962a-cec0233e0288-istiod-ca-cert\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.476482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476413    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99da1e81-f392-47a3-962a-cec0233e0288-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99da1e81-f392-47a3-962a-cec0233e0288-istio-podinfo\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.476544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476486    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-istio-token\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-istio-token\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.476544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476501    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-data" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:34.476544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476531    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-socket\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.476762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476738    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-kube-api-access-mhgdf\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-kube-api-access-mhgdf\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.476840 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476818    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-envoy\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.476840 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476823    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99da1e81-f392-47a3-962a-cec0233e0288-istiod-ca-cert" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:34.476954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.476849    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-certs\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.477150 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.477125    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-socket" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:34.477150 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.477130    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-credential-socket\") pod \"99da1e81-f392-47a3-962a-cec0233e0288\" (UID: \"99da1e81-f392-47a3-962a-cec0233e0288\") "
Oct 31 02:01:34.477623 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.477599    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-credential-socket" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:34.477807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.477782    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-certs" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:34.478403 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.478378    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.478528 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.478513    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.478639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.478626    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.478773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.478759    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99da1e81-f392-47a3-962a-cec0233e0288-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.478936 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.478895    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.479462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.479403    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-istio-token" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:34.479462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.479431    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99da1e81-f392-47a3-962a-cec0233e0288-istio-podinfo" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:34.480172 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.480145    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-kube-api-access-mhgdf" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "kube-api-access-mhgdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:34.481783 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.481748    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-envoy" pod "99da1e81-f392-47a3-962a-cec0233e0288" (UID: "99da1e81-f392-47a3-962a-cec0233e0288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:34.581119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.581086    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhgdf\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-kube-api-access-mhgdf\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.581119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.581115    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99da1e81-f392-47a3-962a-cec0233e0288-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.581119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.581125    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99da1e81-f392-47a3-962a-cec0233e0288-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.581383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.581134    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99da1e81-f392-47a3-962a-cec0233e0288-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:34.667489 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.667457    5293 generic.go:356] "Generic (PLEG): container finished" podID="99da1e81-f392-47a3-962a-cec0233e0288" containerID="421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35" exitCode=0
Oct 31 02:01:34.667911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.667504    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"
Oct 31 02:01:34.667911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.667542    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" event={"ID":"99da1e81-f392-47a3-962a-cec0233e0288","Type":"ContainerDied","Data":"421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35"}
Oct 31 02:01:34.667911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.667593    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l" event={"ID":"99da1e81-f392-47a3-962a-cec0233e0288","Type":"ContainerDied","Data":"be22453cda01f70ba4c8217bfe16893069b27390ccb191ac9ba5c3827f554f47"}
Oct 31 02:01:34.667911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.667618    5293 scope.go:122] "RemoveContainer" containerID="421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35"
Oct 31 02:01:34.668144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.668131    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:34.669108 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.669094    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:01:34.680054 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.680019    5293 scope.go:122] "RemoveContainer" containerID="421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35"
Oct 31 02:01:34.680352 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:01:34.680327    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35\": container with ID starting with 421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35 not found: ID does not exist" containerID="421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35"
Oct 31 02:01:34.680482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.680359    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35"} err="failed to get container status \"421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35\": rpc error: code = NotFound desc = could not find container \"421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35\": container with ID starting with 421f68f6f067783aded2cc5f8a378418f33580df481de549fd9e798238173b35 not found: ID does not exist"
Oct 31 02:01:34.720107 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.714116    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"]
Oct 31 02:01:34.720107 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:34.719611    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cmp7l"]
Oct 31 02:01:36.177332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.177288    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99da1e81-f392-47a3-962a-cec0233e0288" path="/var/lib/kubelet/pods/99da1e81-f392-47a3-962a-cec0233e0288/volumes"
Oct 31 02:01:36.434012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.433927    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"]
Oct 31 02:01:36.434684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.434666    5293 cpu_manager.go:414] "containerMap: removing container" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy"
Oct 31 02:01:36.434684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.434684    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy"
Oct 31 02:01:36.434894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.434842    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="99da1e81-f392-47a3-962a-cec0233e0288" containerName="istio-proxy"
Oct 31 02:01:36.440178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.440155    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:36.448171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.448137    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-28e4f3\""
Oct 31 02:01:36.448298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.448179    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-28e4f3-dockercfg-glt48\""
Oct 31 02:01:36.450303 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.450276    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"]
Oct 31 02:01:36.488642 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.488615    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"]
Oct 31 02:01:36.537597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.537562    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:36.599065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.572759    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbp64\" (UniqueName: \"kubernetes.io/projected/af726e94-0607-4529-a9da-da1d698ff6ca-kube-api-access-hbp64\") pod \"payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s\" (UID: \"af726e94-0607-4529-a9da-da1d698ff6ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:36.796482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.741289    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af726e94-0607-4529-a9da-da1d698ff6ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s\" (UID: \"af726e94-0607-4529-a9da-da1d698ff6ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:36.867116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.867062    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbp64\" (UniqueName: \"kubernetes.io/projected/af726e94-0607-4529-a9da-da1d698ff6ca-kube-api-access-hbp64\") pod \"payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s\" (UID: \"af726e94-0607-4529-a9da-da1d698ff6ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:36.917723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.917684    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a86ec52-6276-48bd-9aa3-bf5ad261165e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h\" (UID: \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:36.960147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.956994    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af726e94-0607-4529-a9da-da1d698ff6ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s\" (UID: \"af726e94-0607-4529-a9da-da1d698ff6ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:36.960147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.957297    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqb5n\" (UniqueName: \"kubernetes.io/projected/0a86ec52-6276-48bd-9aa3-bf5ad261165e-kube-api-access-rqb5n\") pod \"payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h\" (UID: \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:36.960147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.955172    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbp64\" (UniqueName: \"kubernetes.io/projected/af726e94-0607-4529-a9da-da1d698ff6ca-kube-api-access-hbp64\") pod \"payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s\" (UID: \"af726e94-0607-4529-a9da-da1d698ff6ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:36.960147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.958550    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af726e94-0607-4529-a9da-da1d698ff6ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s\" (UID: \"af726e94-0607-4529-a9da-da1d698ff6ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:36.966777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:36.966674    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"]
Oct 31 02:01:37.058624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.058529    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a86ec52-6276-48bd-9aa3-bf5ad261165e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h\" (UID: \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:37.058821 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.058725    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqb5n\" (UniqueName: \"kubernetes.io/projected/0a86ec52-6276-48bd-9aa3-bf5ad261165e-kube-api-access-rqb5n\") pod \"payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h\" (UID: \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:37.059386 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.059346    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a86ec52-6276-48bd-9aa3-bf5ad261165e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h\" (UID: \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:37.062537 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.062518    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:37.070917 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.070879    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqb5n\" (UniqueName: \"kubernetes.io/projected/0a86ec52-6276-48bd-9aa3-bf5ad261165e-kube-api-access-rqb5n\") pod \"payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h\" (UID: \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:37.272755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.272705    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:37.386142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.383744    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"]
Oct 31 02:01:37.388110 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:01:37.385811    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf726e94_0607_4529_a9da_da1d698ff6ca.slice/crio-9ede876f9392fcc53025ac56e2b811c47221180e0465d937051397586b0f3799 WatchSource:0}: Error finding container 9ede876f9392fcc53025ac56e2b811c47221180e0465d937051397586b0f3799: Status 404 returned error can't find the container with id 9ede876f9392fcc53025ac56e2b811c47221180e0465d937051397586b0f3799
Oct 31 02:01:37.548265 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.548226    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"]
Oct 31 02:01:37.553537 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.553513    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"]
Oct 31 02:01:37.553696 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.553641    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:37.558255 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.558219    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:37.561006 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.560903    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-df9d42\""
Oct 31 02:01:37.565993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.565784    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"]
Oct 31 02:01:37.571174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.571148    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"]
Oct 31 02:01:37.576485 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.576446    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872a4172-8fca-4b53-8bb4-aa0ae3010b58-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw\" (UID: \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:37.576593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.576559    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvpd\" (UniqueName: \"kubernetes.io/projected/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-kube-api-access-2dvpd\") pod \"payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh\" (UID: \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:37.576698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.576681    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lfg\" (UniqueName: \"kubernetes.io/projected/872a4172-8fca-4b53-8bb4-aa0ae3010b58-kube-api-access-t8lfg\") pod \"payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw\" (UID: \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:37.576798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.576781    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh\" (UID: \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:37.584742 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.584715    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"]
Oct 31 02:01:37.692067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.688446    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh\" (UID: \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:37.742066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.734243    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh\" (UID: \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:37.742066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.734995    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872a4172-8fca-4b53-8bb4-aa0ae3010b58-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw\" (UID: \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:37.758078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.750085    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872a4172-8fca-4b53-8bb4-aa0ae3010b58-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw\" (UID: \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:37.758078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.750251    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvpd\" (UniqueName: \"kubernetes.io/projected/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-kube-api-access-2dvpd\") pod \"payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh\" (UID: \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:37.758078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.750421    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lfg\" (UniqueName: \"kubernetes.io/projected/872a4172-8fca-4b53-8bb4-aa0ae3010b58-kube-api-access-t8lfg\") pod \"payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw\" (UID: \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:37.806451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.806410    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvpd\" (UniqueName: \"kubernetes.io/projected/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-kube-api-access-2dvpd\") pod \"payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh\" (UID: \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:37.819825 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.819798    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lfg\" (UniqueName: \"kubernetes.io/projected/872a4172-8fca-4b53-8bb4-aa0ae3010b58-kube-api-access-t8lfg\") pod \"payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw\" (UID: \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:37.883961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.883872    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:37.893431 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:37.893396    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:38.110797 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.109644    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s" event={"ID":"af726e94-0607-4529-a9da-da1d698ff6ca","Type":"ContainerStarted","Data":"89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445"}
Oct 31 02:01:38.110797 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.109721    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s" event={"ID":"af726e94-0607-4529-a9da-da1d698ff6ca","Type":"ContainerStarted","Data":"9ede876f9392fcc53025ac56e2b811c47221180e0465d937051397586b0f3799"}
Oct 31 02:01:38.111086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.111032    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:38.116077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.112172    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h" event={"ID":"0a86ec52-6276-48bd-9aa3-bf5ad261165e","Type":"ContainerStarted","Data":"868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61"}
Oct 31 02:01:38.116077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.112219    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h" event={"ID":"0a86ec52-6276-48bd-9aa3-bf5ad261165e","Type":"ContainerStarted","Data":"2501c6f98c24727f7dc3b0f5230e1746f44ef2397e68abf66d5829430a74d296"}
Oct 31 02:01:38.116077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.112955    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:38.135383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.132918    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s" podStartSLOduration=2.132901764 podStartE2EDuration="2.132901764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:38.132364686 +0000 UTC m=+2456.081013079" watchObservedRunningTime="2026-10-31 02:01:38.132901764 +0000 UTC m=+2456.081550167"
Oct 31 02:01:38.668160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.665514    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h" podStartSLOduration=2.665490539 podStartE2EDuration="2.665490539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:38.154757227 +0000 UTC m=+2456.103405619" watchObservedRunningTime="2026-10-31 02:01:38.665490539 +0000 UTC m=+2456.614138933"
Oct 31 02:01:38.669340 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:01:38.669292    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6edfeb_6fb0_4bdd_be9d_6903b9e1da0b.slice/crio-e48c48423168b8e6a788faa1273600695bf863e5de0aef99c0b9c25a66bf9885 WatchSource:0}: Error finding container e48c48423168b8e6a788faa1273600695bf863e5de0aef99c0b9c25a66bf9885: Status 404 returned error can't find the container with id e48c48423168b8e6a788faa1273600695bf863e5de0aef99c0b9c25a66bf9885
Oct 31 02:01:38.673709 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.673634    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"]
Oct 31 02:01:38.680355 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:38.680203    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"]
Oct 31 02:01:39.195444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:39.195382    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh" event={"ID":"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b","Type":"ContainerStarted","Data":"c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268"}
Oct 31 02:01:39.195788 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:39.195754    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh" event={"ID":"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b","Type":"ContainerStarted","Data":"e48c48423168b8e6a788faa1273600695bf863e5de0aef99c0b9c25a66bf9885"}
Oct 31 02:01:39.197323 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:39.197301    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:39.209590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:39.206879    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw" event={"ID":"872a4172-8fca-4b53-8bb4-aa0ae3010b58","Type":"ContainerStarted","Data":"fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73"}
Oct 31 02:01:39.209590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:39.206950    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:39.209590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:39.206974    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw" event={"ID":"872a4172-8fca-4b53-8bb4-aa0ae3010b58","Type":"ContainerStarted","Data":"5ef2780701c293584e1212ab5ac4293d487263063f1da0911feede72ea5d8b43"}
Oct 31 02:01:39.340151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:39.340080    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh" podStartSLOduration=2.340063477 podStartE2EDuration="2.340063477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:39.245660052 +0000 UTC m=+2457.194308445" watchObservedRunningTime="2026-10-31 02:01:39.340063477 +0000 UTC m=+2457.288711871"
Oct 31 02:01:39.378485 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:39.378423    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw" podStartSLOduration=2.378405607 podStartE2EDuration="2.378405607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:39.268570302 +0000 UTC m=+2457.217218697" watchObservedRunningTime="2026-10-31 02:01:39.378405607 +0000 UTC m=+2457.327053999"
Oct 31 02:01:41.779235 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:01:41.779194    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:01:43.489889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:43.489823    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr" podUID="6a8d314b-9d5a-4174-b519-faf043158430" containerName="main" containerID="cri-o://ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc" gracePeriod=45
Oct 31 02:01:43.498670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:43.498487    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr" podUID="6a8d314b-9d5a-4174-b519-faf043158430" containerName="main" probeResult="failure" output="Get \"https://10.133.0.234:8000/ready\": dial tcp 10.133.0.234:8000: connect: connection refused"
Oct 31 02:01:44.113174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.113145    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 02:01:44.176012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.175976    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8d314b-9d5a-4174-b519-faf043158430-kube-api-access-qkvcz\" (UniqueName: \"kubernetes.io/projected/6a8d314b-9d5a-4174-b519-faf043158430-kube-api-access-qkvcz\") pod \"6a8d314b-9d5a-4174-b519-faf043158430\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") "
Oct 31 02:01:44.176221 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.176109    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-tmp-dir\") pod \"6a8d314b-9d5a-4174-b519-faf043158430\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") "
Oct 31 02:01:44.176304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.176243    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a8d314b-9d5a-4174-b519-faf043158430-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8d314b-9d5a-4174-b519-faf043158430-tls-certs\") pod \"6a8d314b-9d5a-4174-b519-faf043158430\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") "
Oct 31 02:01:44.176371 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.176327    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-home\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-home\") pod \"6a8d314b-9d5a-4174-b519-faf043158430\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") "
Oct 31 02:01:44.176433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.176407    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-model-cache\") pod \"6a8d314b-9d5a-4174-b519-faf043158430\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") "
Oct 31 02:01:44.176498 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.176479    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-dshm\") pod \"6a8d314b-9d5a-4174-b519-faf043158430\" (UID: \"6a8d314b-9d5a-4174-b519-faf043158430\") "
Oct 31 02:01:44.176591 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.176563    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-tmp-dir" pod "6a8d314b-9d5a-4174-b519-faf043158430" (UID: "6a8d314b-9d5a-4174-b519-faf043158430"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:44.176729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.176708    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-home" pod "6a8d314b-9d5a-4174-b519-faf043158430" (UID: "6a8d314b-9d5a-4174-b519-faf043158430"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:44.176807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.176772    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-model-cache" pod "6a8d314b-9d5a-4174-b519-faf043158430" (UID: "6a8d314b-9d5a-4174-b519-faf043158430"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:44.178227 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.178208    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:44.178385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.178371    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:44.178545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.178530    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:44.180497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.180444    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-dshm" pod "6a8d314b-9d5a-4174-b519-faf043158430" (UID: "6a8d314b-9d5a-4174-b519-faf043158430"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:44.180697 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.180665    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8d314b-9d5a-4174-b519-faf043158430-tls-certs" pod "6a8d314b-9d5a-4174-b519-faf043158430" (UID: "6a8d314b-9d5a-4174-b519-faf043158430"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:01:44.180799 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.180728    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8d314b-9d5a-4174-b519-faf043158430-kube-api-access-qkvcz" pod "6a8d314b-9d5a-4174-b519-faf043158430" (UID: "6a8d314b-9d5a-4174-b519-faf043158430"). InnerVolumeSpecName "kube-api-access-qkvcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:44.280023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.279956    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8d314b-9d5a-4174-b519-faf043158430-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:44.280023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.279985    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkvcz\" (UniqueName: \"kubernetes.io/projected/6a8d314b-9d5a-4174-b519-faf043158430-kube-api-access-qkvcz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:44.280023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.279994    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8d314b-9d5a-4174-b519-faf043158430-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:44.405320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.405284    5293 generic.go:356] "Generic (PLEG): container finished" podID="6a8d314b-9d5a-4174-b519-faf043158430" containerID="ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc" exitCode=0
Oct 31 02:01:44.405524 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.405325    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"
Oct 31 02:01:44.405524 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.405367    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr" event={"ID":"6a8d314b-9d5a-4174-b519-faf043158430","Type":"ContainerDied","Data":"ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc"}
Oct 31 02:01:44.405524 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.405416    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr" event={"ID":"6a8d314b-9d5a-4174-b519-faf043158430","Type":"ContainerDied","Data":"97e7aba6af32b788e5e78338eadbbf5c4c9fb99fc95c1805bd32ebabdce4016f"}
Oct 31 02:01:44.405524 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.405438    5293 scope.go:122] "RemoveContainer" containerID="ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc"
Oct 31 02:01:44.418633 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.418615    5293 scope.go:122] "RemoveContainer" containerID="ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc"
Oct 31 02:01:44.418947 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:01:44.418926    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc\": container with ID starting with ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc not found: ID does not exist" containerID="ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc"
Oct 31 02:01:44.419134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.418953    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc"} err="failed to get container status \"ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc\": rpc error: code = NotFound desc = could not find container \"ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc\": container with ID starting with ae38e53981043530ebed61153499a356b1da3feb21801496e78addfe6f0a1dfc not found: ID does not exist"
Oct 31 02:01:44.430202 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.430168    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"]
Oct 31 02:01:44.435182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:44.435160    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae5a1c/auth-test-model-ae5a1c-kserve-d89b4d8bf-tdlnr"]
Oct 31 02:01:46.176170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.176142    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a8d314b-9d5a-4174-b519-faf043158430" path="/var/lib/kubelet/pods/6a8d314b-9d5a-4174-b519-faf043158430/volumes"
Oct 31 02:01:46.469987 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.469906    5293 scope.go:122] "RemoveContainer" containerID="311a79b393e19338801f3a6aa6741e1e7db546333785ee82cd06fc8d6f401f4a"
Oct 31 02:01:46.481148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.481124    5293 scope.go:122] "RemoveContainer" containerID="a0e6d241131a3ece5ab7d97628231a8aa1806e29fddec9a0584a3da80728d83a"
Oct 31 02:01:46.491230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.491208    5293 scope.go:122] "RemoveContainer" containerID="fb324563c0f125f68bcf4a060af16bbf3fe0e88e20681d43d55d88564103819a"
Oct 31 02:01:46.502846 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.502821    5293 scope.go:122] "RemoveContainer" containerID="fc5460bb836055da3e7c82ea0e85b1bfab7d6124bf843e8d20f805c83037d3b3"
Oct 31 02:01:46.522639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.522617    5293 scope.go:122] "RemoveContainer" containerID="f1362ec733f294253be05071db71f6642f88f225b745a294839eb384506e591d"
Oct 31 02:01:46.534523 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.534496    5293 scope.go:122] "RemoveContainer" containerID="91b69fb67a443a9ef8d69ccef7509d5f5c26edb3b0b6f303195d648aed1c174c"
Oct 31 02:01:46.546426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.546407    5293 scope.go:122] "RemoveContainer" containerID="80a43f1cc906a6e9b92a07442ea4547f2ad2656b5ce0059c39c7c8443db16ecb"
Oct 31 02:01:46.568092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.568068    5293 scope.go:122] "RemoveContainer" containerID="5f07e899d95f8d4dbe4a2905433aff3fd337a83d78bc3f2d1d8de254e67f9f47"
Oct 31 02:01:46.631973 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:46.631852    5293 scope.go:122] "RemoveContainer" containerID="258862eb059a019f8a96af7e7b23c1015f14492ce5e6a52d3ae8a7bdcc0b2f80"
Oct 31 02:01:48.146290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.146263    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"]
Oct 31 02:01:48.160727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.158259    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6a8d314b-9d5a-4174-b519-faf043158430" containerName="main"
Oct 31 02:01:48.160727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.158288    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8d314b-9d5a-4174-b519-faf043158430" containerName="main"
Oct 31 02:01:48.160727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.158488    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8d314b-9d5a-4174-b519-faf043158430" containerName="main"
Oct 31 02:01:48.174833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.168626    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"]
Oct 31 02:01:48.174833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.168798    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.175735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.175714    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"]
Oct 31 02:01:48.176103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.175868    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.212787 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.212763    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"]
Oct 31 02:01:48.256095 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.256057    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-91b6b1-openshift-default-dockercfg-9wq4c\""
Oct 31 02:01:48.295395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295366    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.295580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295403    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.295580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295440    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.295718 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295594    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.295718 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295617    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.295718 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295639    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.295718 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295666    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.295852 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295776    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h89nv\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-kube-api-access-h89nv\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.295852 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295797    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.295852 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295836    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.295949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295895    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.295949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295914    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.295949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295930    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nhwz\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-kube-api-access-8nhwz\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.296087 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.295972    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.296209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.296182    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.296295 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.296268    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.296361 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.296319    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.296468 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.296451    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.398426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398336    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.398426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398382    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.398426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398417    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.398777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398457    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.398777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398481    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.398777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398593    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.398777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398669    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.398777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398736    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89nv\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-kube-api-access-h89nv\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.398777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398771    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.399129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398823    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.399129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398839    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.399129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398924    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.399129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.398964    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.399129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.399007    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nhwz\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-kube-api-access-8nhwz\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.399412 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.399172    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.399412 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.399323    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.399560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.399525    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.399799 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.399658    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.399799 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.399750    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.399916 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.399810    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.400704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.400138    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.400704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.400151    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.400704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.400189    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.400704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.400404    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.400704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.400646    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.401462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.399835    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.401809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.401678    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.402061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.402018    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.402180 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.402158    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.402875 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.402855    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.402949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.402856    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.402949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.402913    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.416474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.416450    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.416936 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.416913    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.423556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.423513    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89nv\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-kube-api-access-h89nv\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.423949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.423930    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nhwz\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-kube-api-access-8nhwz\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.515854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.515815    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:48.518942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.518911    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:48.832502 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:01:48.832456    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162bf7e4_3a15_4b30_a9c6_f816a6327a12.slice/crio-33e817ab969c82742480fc10e553f56220b65d7ebc59dbf37fdd7fbd59392b54 WatchSource:0}: Error finding container 33e817ab969c82742480fc10e553f56220b65d7ebc59dbf37fdd7fbd59392b54: Status 404 returned error can't find the container with id 33e817ab969c82742480fc10e553f56220b65d7ebc59dbf37fdd7fbd59392b54
Oct 31 02:01:48.836439 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.836417    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"]
Oct 31 02:01:48.836846 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:01:48.836770    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdf2c36_fc81_44e6_b79b_4b6ccd770189.slice/crio-f9a2e421048642f36a0f7c0dfa569616ea025d2805aadadd6b121ccccc5c6ce5 WatchSource:0}: Error finding container f9a2e421048642f36a0f7c0dfa569616ea025d2805aadadd6b121ccccc5c6ce5: Status 404 returned error can't find the container with id f9a2e421048642f36a0f7c0dfa569616ea025d2805aadadd6b121ccccc5c6ce5
Oct 31 02:01:48.836999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.836857    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:48.837132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.837105    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:48.837188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.837171    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:48.841699 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.841647    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"]
Oct 31 02:01:48.841809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.841691    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:48.841809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.841764    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:48.841909 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:48.841809    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:49.509600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:49.509548    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" event={"ID":"9cdf2c36-fc81-44e6-b79b-4b6ccd770189","Type":"ContainerStarted","Data":"2cb1a8a685caea5438f733adf141ef4c3db961a4c487f1489d4ef2208afb7bd5"}
Oct 31 02:01:49.510141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:49.509662    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" event={"ID":"9cdf2c36-fc81-44e6-b79b-4b6ccd770189","Type":"ContainerStarted","Data":"f9a2e421048642f36a0f7c0dfa569616ea025d2805aadadd6b121ccccc5c6ce5"}
Oct 31 02:01:49.511640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:49.511614    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" event={"ID":"162bf7e4-3a15-4b30-a9c6-f816a6327a12","Type":"ContainerStarted","Data":"a580021edf03119adaf7527a7876de0e89b6762c460417a138db5374150c8793"}
Oct 31 02:01:49.511782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:49.511664    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" event={"ID":"162bf7e4-3a15-4b30-a9c6-f816a6327a12","Type":"ContainerStarted","Data":"33e817ab969c82742480fc10e553f56220b65d7ebc59dbf37fdd7fbd59392b54"}
Oct 31 02:01:49.516182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:49.516158    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:49.519801 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:49.519780    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:49.548760 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:49.548679    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podStartSLOduration=1.5486578560000002 podStartE2EDuration="1.548657856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:49.540802078 +0000 UTC m=+2467.489450479" watchObservedRunningTime="2026-10-31 02:01:49.548657856 +0000 UTC m=+2467.497306248"
Oct 31 02:01:49.571936 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:49.571880    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podStartSLOduration=1.571864187 podStartE2EDuration="1.571864187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:49.569194221 +0000 UTC m=+2467.517842613" watchObservedRunningTime="2026-10-31 02:01:49.571864187 +0000 UTC m=+2467.520512557"
Oct 31 02:01:50.329486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:50.329315    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:01:50.374138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:50.373201    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:01:50.395198 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:50.394229    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:01:50.524880 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:50.524838    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 31 02:01:50.525601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:50.525553    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 31 02:01:50.561398 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:50.561356    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 02:01:50.562145 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:50.562099    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 02:01:51.059064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.032495    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"]
Oct 31 02:01:51.059064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.046472    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.059064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.054595    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"agc-cred-8150f38250de\""
Oct 31 02:01:51.059064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.054678    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"routing-overlay\""
Oct 31 02:01:51.059064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.054768    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"kube-root-ca.crt\""
Oct 31 02:01:51.059064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.055508    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"payload-processing-external-model-e2e--e5c46109-dockercfg-4f6vp\""
Oct 31 02:01:51.059064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.055587    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"payload-processing-external-model-plugins-e2e-extmod-b-91b6b1\""
Oct 31 02:01:51.076172 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.076132    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"]
Oct 31 02:01:51.186066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.169205    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"openshift-service-ca.crt\""
Oct 31 02:01:51.194062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.190599    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.194062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.190731    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.194062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.190875    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.233065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.205228    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-kube-api-access-w5sdf\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.330117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.322254    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:01:51.407083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.403407    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.407083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.403585    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-kube-api-access-w5sdf\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.407083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.403717    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.413061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.411251    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.413061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.412012    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.439070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.437169    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.459067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.446023    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.561062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.549175    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-kube-api-access-w5sdf\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.585120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.573754    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 31 02:01:51.585120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.573818    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 31 02:01:51.585120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.574670    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 02:01:51.585120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.574749    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 02:01:51.706656 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.694333    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:51.981353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:51.981317    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"]
Oct 31 02:01:52.049249 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.037264    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"]
Oct 31 02:01:52.049249 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.044186    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.069066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.061657    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bb5c51-openshift-default-dockercfg-s2hdn\""
Oct 31 02:01:52.182078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.179909    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.268350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.268311    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.290475 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.269823    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.291254 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.291221    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.293521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.293490    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.310433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.310401    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.310643 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.310626    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.314059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.310767    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.382221 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.382179    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9w5\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-kube-api-access-fk9w5\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.382454 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.382433    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.382579 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.382563    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.382835 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.382814    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.382966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.382951    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.383102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.383085    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.383278 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.383262    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.383386 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.383371    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.383615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.383597    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.383754 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.383739    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.383920 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.383902    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9nmt\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-kube-api-access-g9nmt\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.486963 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.486874    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.490829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.487503    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.491051 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.490876    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.491344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.491306    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.491792 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.491756    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.491894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.491809    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.491894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.491854    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.492016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.491892    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.492016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.491969    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9w5\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-kube-api-access-fk9w5\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.492016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492009    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.492200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492056    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.492200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492145    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.492318 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492190    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.492318 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492234    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.492318 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492305    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.492472 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492340    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.492527 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492471    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.492580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492538    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.492637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492601    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9nmt\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-kube-api-access-g9nmt\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.492848 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.492831    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.493362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.493338    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.493619 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.493602    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.494239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.494187    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.494732    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"]
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.494803    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"]
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.494813    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.495300    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.495569    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.495829    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.495978    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.509331    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.567168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.547180    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9w5\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-kube-api-access-fk9w5\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.568529 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.568475    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.580681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.578022    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 02:01:52.580681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.578122    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 31 02:01:52.580681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.578121    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 02:01:52.580681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.578213    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 31 02:01:52.580681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.578470    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.580681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.579983    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.581063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.580722    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.582787 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.582735    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9nmt\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-kube-api-access-g9nmt\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:52.584809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.584571    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.588724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.588705    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"]
Oct 31 02:01:52.785539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.784213    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:52.872226 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:52.872182    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:53.289856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.289683    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j" event={"ID":"bf575678-3aed-403c-8a18-8e94699edbac","Type":"ContainerStarted","Data":"e5966c3487a3e38cd47288e2f0d62349baa2724ee5b220ed537e4506fb1f1ae4"}
Oct 31 02:01:53.289856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.289744    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j" event={"ID":"bf575678-3aed-403c-8a18-8e94699edbac","Type":"ContainerStarted","Data":"3b73e3e174abefd1d68fe357d035794abcf028d3d1823e50c91e7a8133ca9e57"}
Oct 31 02:01:53.290579 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.290538    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:01:53.574541 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.574503    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 02:01:53.625471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.625372    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 02:01:53.667241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.660587    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 31 02:01:53.667241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.660673    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 31 02:01:53.678083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.678015    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j" podStartSLOduration=3.677999587 podStartE2EDuration="3.677999587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:53.325695648 +0000 UTC m=+2471.274344066" watchObservedRunningTime="2026-10-31 02:01:53.677999587 +0000 UTC m=+2471.626647981"
Oct 31 02:01:53.678996 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.678970    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"]
Oct 31 02:01:53.682846 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:01:53.682821    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afd0a65_9e74_4835_a35a_72b63c08c7a8.slice/crio-7731323cc4b5c02dc6eea75968492504f8a740e836fb237e406c4d99537b1501 WatchSource:0}: Error finding container 7731323cc4b5c02dc6eea75968492504f8a740e836fb237e406c4d99537b1501: Status 404 returned error can't find the container with id 7731323cc4b5c02dc6eea75968492504f8a740e836fb237e406c4d99537b1501
Oct 31 02:01:53.687325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.686537    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:53.687325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.686620    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:53.687325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.686653    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:53.752415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.752376    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"]
Oct 31 02:01:53.769932 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:01:53.769347    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ac2bc5_bbbe_4af0_91c6_02bad5837a75.slice/crio-968c2d7df8b19a3a3630fac5a2985625ae78452c241c0535ddddfc16a64f9882 WatchSource:0}: Error finding container 968c2d7df8b19a3a3630fac5a2985625ae78452c241c0535ddddfc16a64f9882: Status 404 returned error can't find the container with id 968c2d7df8b19a3a3630fac5a2985625ae78452c241c0535ddddfc16a64f9882
Oct 31 02:01:53.774408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.774319    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:53.774516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.774432    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:53.774516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:53.774479    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:01:54.547072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.525610    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 02:01:54.547072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.525663    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 02:01:54.549657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.547506    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 31 02:01:54.549657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.547565    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 31 02:01:54.581618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.581574    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" event={"ID":"9afd0a65-9e74-4835-a35a-72b63c08c7a8","Type":"ContainerStarted","Data":"e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30"}
Oct 31 02:01:54.582188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.582165    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" event={"ID":"9afd0a65-9e74-4835-a35a-72b63c08c7a8","Type":"ContainerStarted","Data":"7731323cc4b5c02dc6eea75968492504f8a740e836fb237e406c4d99537b1501"}
Oct 31 02:01:54.614124 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.614081    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" event={"ID":"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75","Type":"ContainerStarted","Data":"968c2d7df8b19a3a3630fac5a2985625ae78452c241c0535ddddfc16a64f9882"}
Oct 31 02:01:54.697071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.694099    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podStartSLOduration=3.694076295 podStartE2EDuration="3.694076295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:54.618369232 +0000 UTC m=+2472.567017626" watchObservedRunningTime="2026-10-31 02:01:54.694076295 +0000 UTC m=+2472.642724688"
Oct 31 02:01:54.703777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.699388    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podStartSLOduration=3.69937574 podStartE2EDuration="3.69937574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:54.646390243 +0000 UTC m=+2472.595038637" watchObservedRunningTime="2026-10-31 02:01:54.69937574 +0000 UTC m=+2472.648024136"
Oct 31 02:01:54.792879 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.792562    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:01:54.903063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:54.902520    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:01:55.519588 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.519551    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 02:01:55.519872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.519840    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 02:01:55.520599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.520575    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 31 02:01:55.530182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.530134    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 31 02:01:55.765067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.764113    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" event={"ID":"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75","Type":"ContainerStarted","Data":"19e5a8fff6e3717eae00cd740ae2c8fd4d1f2054c5d9bd714b7d1715acaf9f72"}
Oct 31 02:01:55.790194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.790102    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 31 02:01:55.790483 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.790433    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 31 02:01:55.919072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.918797    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 02:01:55.919072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:55.918853    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 02:01:56.521704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.520692    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 31 02:01:56.521704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.520768    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 31 02:01:56.521704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.521584    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 02:01:56.521704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.521634    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 02:01:56.732066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.725687    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:56.854388 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.854288    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 31 02:01:56.889064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.879830    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:56.889064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.879541    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 31 02:01:56.905201 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.893377    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 02:01:56.905559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.905514    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 02:01:56.979730 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.978228    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:56.979730 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:56.978873    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:57.076068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.075132    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:57.080061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.076295    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:57.102065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.097123    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:57.102065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.097528    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:57.445063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.444982    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:57.516653 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.516619    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 31 02:01:57.516984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.516938    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 31 02:01:57.521500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.521474    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 02:01:57.521698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.521672    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 02:01:57.531796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.531759    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:57.759123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.727685    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:01:57.759123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.728407    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:01:57.902554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.888544    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 31 02:01:57.902554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.888609    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 31 02:01:57.902554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.892216    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:01:57.967065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.910553    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:01:57.967065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.911113    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:01:57.967065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.925716    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 02:01:57.967065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.925765    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 02:01:57.996078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.982192    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:57.996078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.995235    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:01:57.996078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.995684    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:01:57.996078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:57.995742    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:01:58.116201 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.106140    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:58.163474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.145485    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:01:58.233082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.219410    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:01:58.509297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.475128    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:01:58.551723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.545117    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:01:58.730293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.730250    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:58.732772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.732747    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:01:58.733376 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.732975    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:01:58.733561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.733548    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:01:58.794387 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.794301    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 31 02:01:58.795324 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.795299    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 31 02:01:58.795507 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.795486    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 31 02:01:58.911494 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.911444    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:01:58.913972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.913947    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:01:58.927351 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.927296    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 02:01:58.927658 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.927602    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 02:01:59.004941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:58.993539    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:59.113345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.113215    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:59.114424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.114391    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:59.239901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.234614    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:59.295981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.295925    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:59.309963 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.309861    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:01:59.346441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.346383    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:59.515843 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.515802    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:59.536496 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.536463    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:59.553234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.553208    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:59.554735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.554714    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:59.554941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.552302    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:01:59.570179 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.562446    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:01:59.581162 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.581136    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:01:59.691114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.689180    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:59.763067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.762118    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:01:59.834146 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.829533    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 31 02:01:59.834146 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.829592    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 31 02:01:59.930182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.930137    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:01:59.996255 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.996211    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:59.997721 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:01:59.997682    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:00.028587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.028550    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 02:02:00.028870 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.028838    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 02:02:00.167129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.159530    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:02:00.188111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.151150    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:02:00.276544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.276508    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:02:00.276745 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.276596    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:02:00.288140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.288067    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:02:00.290535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.290513    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:02:00.292311 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.292290    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:02:00.293555 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.293535    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:02:00.308492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.308465    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:02:00.316246 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.316220    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:02:00.791310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.791276    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 31 02:02:00.791678 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.791645    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 31 02:02:00.872858 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.872828    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 02:02:00.873135 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:00.873110    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 02:02:01.785191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:01.785154    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 31 02:02:01.785570 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:01.785214    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 31 02:02:01.872740 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:01.872704    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 02:02:01.872933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:01.872788    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 02:02:02.784929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:02.784897    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 31 02:02:02.785156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:02.784947    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 31 02:02:02.875469 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:02.875434    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 02:02:02.876122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:02.876091    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 02:02:03.164378 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.164305    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"]
Oct 31 02:02:03.193063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.189574    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.195582 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.195537    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"]
Oct 31 02:02:03.267621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.267588    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.275090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.272940    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"]
Oct 31 02:02:03.282360 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.282332    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"]
Oct 31 02:02:03.331345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.331301    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.332246 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.331728    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.332593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.332569    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.333857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.333836    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.334391 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.334372    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.334595 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.334578    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.334785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.334767    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwfw\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-kube-api-access-gnwfw\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.348429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.348387    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.348810 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.348790    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.451061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.450953    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.451061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451027    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.451307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451073    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.451307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451098    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.451307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451132    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.451307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451273    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87139ff5-903f-424b-9e99-b8740dc075e2-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.451541 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451398    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52k2n\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-kube-api-access-52k2n\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.451541 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451454    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.451655 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451546    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.451655 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451605    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.451765 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451724    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.451765 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451753    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.451849 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451780    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.451849 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451833    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.451964 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451923    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.452029 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451965    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.452029 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.451986    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwfw\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-kube-api-access-gnwfw\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.452178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.452053    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87139ff5-903f-424b-9e99-b8740dc075e2-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.452178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.452101    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.452178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.452107    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.452363 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.452293    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.452586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.452558    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.452662 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.452612    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.454303 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.454276    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.454405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.454348    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.462637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.462614    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.463540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.463516    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwfw\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-kube-api-access-gnwfw\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.553617 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.553578    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52k2n\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-kube-api-access-52k2n\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.553808 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.553678    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.553808 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.553794    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.553897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.553841    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.553934 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.553903    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87139ff5-903f-424b-9e99-b8740dc075e2-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.554186 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.554159    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.554306 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.554193    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.554306 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.554221    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.554306 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.554260    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.554306 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.554292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87139ff5-903f-424b-9e99-b8740dc075e2-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.554613 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.554584    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.554677 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.554616    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.554677 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.554626    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.555199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.555173    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87139ff5-903f-424b-9e99-b8740dc075e2-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.556649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.556625    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.557166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.557143    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87139ff5-903f-424b-9e99-b8740dc075e2-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.565478 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.565453    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52k2n\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-kube-api-access-52k2n\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.565791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.565766    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.573814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.573789    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:03.599708 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.599657    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:03.826077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.822516    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:02:03.826077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.824491    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:02:03.826077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.825385    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:02:03.881838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.881791    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:02:03.883399 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.883378    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:02:03.884824 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:03.884766    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:02:04.027077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.027053    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"]
Oct 31 02:02:04.029869 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:02:04.029835    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc6ec04_1cc3_4859_a53b_10fc82ff3433.slice/crio-9491c650a97a6ac3253b23c768772b144f10e50081340ccd666d88d9bbfbcf6b WatchSource:0}: Error finding container 9491c650a97a6ac3253b23c768772b144f10e50081340ccd666d88d9bbfbcf6b: Status 404 returned error can't find the container with id 9491c650a97a6ac3253b23c768772b144f10e50081340ccd666d88d9bbfbcf6b
Oct 31 02:02:04.033640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.033600    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:04.033735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.033689    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:04.033783 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.033734    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:04.095555 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:02:04.095523    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87139ff5_903f_424b_9e99_b8740dc075e2.slice/crio-844c1e49f16c072f90fff368278e4e81d0e435ccf06ee6a68ffae91b9c6c707d WatchSource:0}: Error finding container 844c1e49f16c072f90fff368278e4e81d0e435ccf06ee6a68ffae91b9c6c707d: Status 404 returned error can't find the container with id 844c1e49f16c072f90fff368278e4e81d0e435ccf06ee6a68ffae91b9c6c707d
Oct 31 02:02:04.095872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.095850    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"]
Oct 31 02:02:04.098933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.098898    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:04.099057 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.098968    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:04.099057 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.098999    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:04.604299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.604273    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:02:04.652393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.652353    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" event={"ID":"87139ff5-903f-424b-9e99-b8740dc075e2","Type":"ContainerStarted","Data":"2142d3f0f394839a8e618233859f769c0df786f17a8f39454cd315747b2b8966"}
Oct 31 02:02:04.652723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.652696    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" event={"ID":"87139ff5-903f-424b-9e99-b8740dc075e2","Type":"ContainerStarted","Data":"844c1e49f16c072f90fff368278e4e81d0e435ccf06ee6a68ffae91b9c6c707d"}
Oct 31 02:02:04.664543 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.664519    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" event={"ID":"0cc6ec04-1cc3-4859-a53b-10fc82ff3433","Type":"ContainerStarted","Data":"ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b"}
Oct 31 02:02:04.664811 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.664790    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" event={"ID":"0cc6ec04-1cc3-4859-a53b-10fc82ff3433","Type":"ContainerStarted","Data":"9491c650a97a6ac3253b23c768772b144f10e50081340ccd666d88d9bbfbcf6b"}
Oct 31 02:02:04.700110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:04.699991    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podStartSLOduration=1.699971261 podStartE2EDuration="1.699971261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:04.686538602 +0000 UTC m=+2482.635186994" watchObservedRunningTime="2026-10-31 02:02:04.699971261 +0000 UTC m=+2482.648619655"
Oct 31 02:02:05.574348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:05.574311    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:05.575752 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:05.575729    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 31 02:02:05.575870 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:05.575785    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 31 02:02:05.600564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:05.600533    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:05.602379 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:05.602317    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 02:02:05.602524 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:05.602408    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 02:02:06.574883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:06.574563    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 31 02:02:06.574883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:06.574637    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 31 02:02:06.602065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:06.601326    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 02:02:06.602065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:06.601401    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 02:02:06.965875 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:06.965797    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podStartSLOduration=3.965776783 podStartE2EDuration="3.965776783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:04.714418357 +0000 UTC m=+2482.663066760" watchObservedRunningTime="2026-10-31 02:02:06.965776783 +0000 UTC m=+2484.914425177"
Oct 31 02:02:06.967358 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:06.967307    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"]
Oct 31 02:02:06.982357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:06.978255    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.040966    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.041149    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmq2n\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-kube-api-access-xmq2n\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.041230    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.041277    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.041304    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.041414    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.041440    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.041553    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.069110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.041632    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df5a77-8db8-41cb-866d-494e7832c0d1-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.078779 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.078750    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"]
Oct 31 02:02:07.166521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.166472    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df5a77-8db8-41cb-866d-494e7832c0d1-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.166729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.166630    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.166804 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.166758    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmq2n\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-kube-api-access-xmq2n\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.166864 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.166830    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.166929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.166880    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.166929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.166909    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.167085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.167018    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.167085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.167065    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.167217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.167176    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.169393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.169351    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df5a77-8db8-41cb-866d-494e7832c0d1-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.169666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.169642    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.169748 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.169721    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.169934 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.169916    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.170542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.170182    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.172000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.171979    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.172584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.172554    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.179363 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.179343    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.181437 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.181416    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmq2n\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-kube-api-access-xmq2n\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.343114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.342690    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:07.579396 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.575698    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 31 02:02:07.579396 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.575768    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 31 02:02:07.653201 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.633687    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 02:02:07.653496 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.653455    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 02:02:07.993526 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:07.992916    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"]
Oct 31 02:02:07.994824 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:02:07.994792    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10df5a77_8db8_41cb_866d_494e7832c0d1.slice/crio-34859db7a97f3e635ece9fce689fcfa41d9c4e24a0e40c86ed12c08b979c6382 WatchSource:0}: Error finding container 34859db7a97f3e635ece9fce689fcfa41d9c4e24a0e40c86ed12c08b979c6382: Status 404 returned error can't find the container with id 34859db7a97f3e635ece9fce689fcfa41d9c4e24a0e40c86ed12c08b979c6382
Oct 31 02:02:08.035107 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.014678    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:08.035107 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.014779    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:08.035107 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.014828    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:08.574768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.574741    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 31 02:02:08.574910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.574811    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 31 02:02:08.626070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.620463    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 02:02:08.626070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.620520    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 02:02:08.784676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.784593    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" event={"ID":"10df5a77-8db8-41cb-866d-494e7832c0d1","Type":"ContainerStarted","Data":"6f8a637ae81c129e39490843a1575424a7fea8eea6785e083782135fc3d5ed84"}
Oct 31 02:02:08.784676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.784647    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" event={"ID":"10df5a77-8db8-41cb-866d-494e7832c0d1","Type":"ContainerStarted","Data":"34859db7a97f3e635ece9fce689fcfa41d9c4e24a0e40c86ed12c08b979c6382"}
Oct 31 02:02:08.817067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:08.816283    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podStartSLOduration=2.816264395 podStartE2EDuration="2.816264395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:08.81367163 +0000 UTC m=+2486.762320024" watchObservedRunningTime="2026-10-31 02:02:08.816264395 +0000 UTC m=+2486.764912788"
Oct 31 02:02:09.348077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:09.347695    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:09.348077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:09.347811    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 02:02:09.348077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:09.347855    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 02:02:09.574573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:09.574539    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 31 02:02:09.574752 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:09.574595    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 31 02:02:09.620117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:09.612300    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 02:02:09.620117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:09.612371    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 02:02:10.357068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:10.353421    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 02:02:10.357068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:10.353502    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 02:02:10.585067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:10.576431    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 31 02:02:10.585067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:10.576505    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 31 02:02:10.604068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:10.601448    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 02:02:10.604068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:10.601500    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 02:02:11.361064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:11.360353    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 02:02:11.361064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:11.360411    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 02:02:11.587083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:11.575421    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 31 02:02:11.587083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:11.575496    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 31 02:02:11.621152 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:11.611792    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 02:02:11.621152 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:11.611853    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 02:02:11.952224 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:02:11.948274    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:02:12.016220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.016165    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:12.349456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.349364    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 02:02:12.349744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.349714    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 02:02:12.656472 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.656432    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:12.658475 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.658428    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:12.659543 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.659512    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:02:12.733065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.729575    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:12.733065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.731638    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:12.735491 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:12.735466    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:02:13.364064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:13.360612    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 02:02:13.364064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:13.360667    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 02:02:14.358064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:14.345211    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 02:02:14.358064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:14.345275    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 02:02:15.344737 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.344660    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 02:02:15.344737 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.344716    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 02:02:15.860069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.855346    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"]
Oct 31 02:02:15.964442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.964410    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"]
Oct 31 02:02:15.966281 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.966253    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:15.970099 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.970080    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"]
Oct 31 02:02:15.970251 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.970227    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:15.970366 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.970242    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"]
Oct 31 02:02:15.973575 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:15.973548    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2186c3-openshift-default-dockercfg-pdnvx\""
Oct 31 02:02:16.089631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.089584    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.089631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.089635    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.089823 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.089740    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.089823 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.089787    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.089953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.089872    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.089953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.089908    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.089953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.089941    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.090170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090067    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77lv\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-kube-api-access-s77lv\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.090170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090140    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.090288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090208    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0a3e1-d848-4931-8bec-bea42f843751-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.090288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090236    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.090367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090318    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.090367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090340    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.090481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090387    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdvx\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-kube-api-access-2tdvx\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.090481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090470    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.090600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090550    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.090600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090588    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0a3e1-d848-4931-8bec-bea42f843751-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.090704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.090691    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196259    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196345    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196428    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196486    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196538    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196594    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196661    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196816    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77lv\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-kube-api-access-s77lv\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.196950    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.197180    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0a3e1-d848-4931-8bec-bea42f843751-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.219067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.197232    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.220551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.220513    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.221299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.221275    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.221761 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.221735    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.241192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.231716    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.241192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.234067    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.241192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.237513    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.241192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.238890    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.269065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.262141    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.269065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.262225    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.270742    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0a3e1-d848-4931-8bec-bea42f843751-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.270955    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.283141    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdvx\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-kube-api-access-2tdvx\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.283236    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.283369    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.283433    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0a3e1-d848-4931-8bec-bea42f843751-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.283585    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.284384    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.285094    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.290065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.286303    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0a3e1-d848-4931-8bec-bea42f843751-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.350081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.348501    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 02:02:16.350081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.348572    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 02:02:16.350081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.348996    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.350081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.349594    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77lv\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-kube-api-access-s77lv\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:16.350465 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.350184    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdvx\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-kube-api-access-2tdvx\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.350465 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.350381    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.355141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.350971    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.355141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.353567    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.587158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.587032    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:16.608087 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:16.608059    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:17.364144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.364116    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"]
Oct 31 02:02:17.381759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.381728    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:17.390864 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.385425    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:17.390864 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.389164    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:02:17.395287 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.395250    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"]
Oct 31 02:02:17.395781 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.395748    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:17.395854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.395831    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:17.395912 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.395870    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:17.399902 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:02:17.399875    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b0a3e1_d848_4931_8bec_bea42f843751.slice/crio-99df0fa9af20a975d55ef51bcd0f476c13c122afc9d4a18b24df584dae7c8b49 WatchSource:0}: Error finding container 99df0fa9af20a975d55ef51bcd0f476c13c122afc9d4a18b24df584dae7c8b49: Status 404 returned error can't find the container with id 99df0fa9af20a975d55ef51bcd0f476c13c122afc9d4a18b24df584dae7c8b49
Oct 31 02:02:17.419588 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.419555    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:17.419798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.419781    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:17.419882 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.419870    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:17.569578 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:17.569546    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:02:18.500108 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.497999    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" event={"ID":"d367975b-ff2a-4a3a-9cd3-a5cd32820603","Type":"ContainerStarted","Data":"fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4"}
Oct 31 02:02:18.500108 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.498737    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" event={"ID":"d367975b-ff2a-4a3a-9cd3-a5cd32820603","Type":"ContainerStarted","Data":"23971f19af819e2c97443e4420f40613522f7c2a9936a157ca76f3e822c58178"}
Oct 31 02:02:18.504235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.504182    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" event={"ID":"57b0a3e1-d848-4931-8bec-bea42f843751","Type":"ContainerStarted","Data":"d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb"}
Oct 31 02:02:18.504440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.504238    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" event={"ID":"57b0a3e1-d848-4931-8bec-bea42f843751","Type":"ContainerStarted","Data":"99df0fa9af20a975d55ef51bcd0f476c13c122afc9d4a18b24df584dae7c8b49"}
Oct 31 02:02:18.536057 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.535989    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podStartSLOduration=3.535970272 podStartE2EDuration="3.535970272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:18.529235309 +0000 UTC m=+2496.477883698" watchObservedRunningTime="2026-10-31 02:02:18.535970272 +0000 UTC m=+2496.484618665"
Oct 31 02:02:18.559224 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.559161    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podStartSLOduration=3.559141428 podStartE2EDuration="3.559141428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:18.551914344 +0000 UTC m=+2496.500562740" watchObservedRunningTime="2026-10-31 02:02:18.559141428 +0000 UTC m=+2496.507789821"
Oct 31 02:02:18.587371 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.587322    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:18.588746 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.588722    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:18.588900 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.588764    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:18.608814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.608780    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:18.610573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.610544    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:18.610692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:18.610598    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:19.599067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:19.591405    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:19.599067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:19.591468    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:19.615066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:19.608441    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:19.615066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:19.608508    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:20.124069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.123119    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.218409 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.216169    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.268194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.266410    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 02:02:20.313065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.310180    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.313065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.311107    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.313065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.311864    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.323065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.320553    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.323065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.321090    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:02:20.589070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.587400    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:20.589070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.587469    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:20.615500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.615458    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:20.616072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.616019    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:20.735602 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.735544    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.870287 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.870162    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:02:20.920741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:20.918172    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:21.009064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.004470    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:21.101068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.098159    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded"
Oct 31 02:02:21.137115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.136194    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:21.137115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.136817    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:02:21.234065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.226094    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:02:21.234065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.226629    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:02:21.479856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.479803    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:02:21.597370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.580149    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:02:21.597370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.580474    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:02:21.597370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.580709    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:02:21.597370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.580937    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.609482    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.609561    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.609603    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.609648    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.609683    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.612610    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.612654    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.643528    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:21.656476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.643595    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:21.736089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.735975    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:21.737140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.737118    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:02:21.937080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.935148    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:21.937080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.936977    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:02:22.035335 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.963109    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.035335 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:21.977536    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:02:22.193417 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.193352    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.243150 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.243102    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.597483 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.597438    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.623246 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.623195    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.625337 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.625298    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.626685 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.626657    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.674067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.645928    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:22.805105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.710211    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:22.889108 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.888959    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:22.889381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.822188    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.948070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.928969    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:22.956066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.951605    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:22.956066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.822146    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:22.956066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.889809    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:02:22.984396 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.974462    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:02:23.025070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:22.999595    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:02:23.215068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.214862    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:23.259447 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.255894    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:23.282970 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.271133    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:23.495357 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:02:23.495186    5293 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.273s"
Oct 31 02:02:23.495673 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.495652    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:02:23.495929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.495912    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:02:23.592488 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.592452    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:23.592799 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.592769    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:23.597918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.597881    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:23.622465 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.622403    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:23.622654 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.622470    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:23.654065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.652180    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:02:23.654065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.652559    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:02:23.654065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.652828    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:02:23.917132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.889694    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:23.933258 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.932249    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:23.965336 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.965299    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:02:23.975171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:23.975132    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:24.053499 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.039458    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:24.226116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.218193    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:24.286746 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.265347    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:24.286746 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.286205    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:02:24.376873 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.376839    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:02:24.758072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.746544    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:02:24.762110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.758705    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:02:24.774881 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.768478    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:02:24.776989 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.774879    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:24.819140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.795243    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:02:24.819140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.803840    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:24.819140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.803938    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:24.819140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.808455    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:24.819140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.808512    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:24.947151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:24.947109    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.062112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.060987    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:02:25.087067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.062358    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.098388 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.098345    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.222090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.221019    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.271325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.271273    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 02:02:25.273218 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.273184    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.289590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.289529    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.590065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.589249    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:25.590065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.589315    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:25.619601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.619561    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:25.619889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.619862    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:25.768064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.763721    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.772170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.769300    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:02:25.787241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.784755    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.810072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.797147    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.986081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:25.969661    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.067294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.066186    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.105431 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.105386    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.133101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.127069    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.228666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.228588    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:02:26.290347 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.290245    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.311975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.311922    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:02:26.597943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.597835    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:26.598258 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.598210    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:26.608947 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.608915    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:26.609267 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.609209    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:26.778301 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.778248    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.779365 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.779324    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.808319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.808265    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:02:26.809372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.809332    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.997105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:26.969869    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:27.084432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.084379    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:02:27.129065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.109196    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:27.142089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.131780    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:27.238597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.238541    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:27.304511 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.304404    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:27.335805 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.315577    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:27.799493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.799449    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:02:27.924113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.799857    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:02:27.924113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.799951    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:27.924113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.919950    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:27.924113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.823406    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:02:27.925096 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.891459    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:27.925594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.925515    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:27.925896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:27.823365    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.014421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.002879    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:02:28.117783 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.116850    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:02:28.144064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.128692    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 02:02:28.144064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.128815    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.144064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.136075    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.262192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.262142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.344883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.344842    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 02:02:28.483357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.347327    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:02:28.489145 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.462795    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:02:28.590367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.590331    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:28.590656 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.590632    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:28.630143 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.630103    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:28.630409 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.630385    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:28.824424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.824147    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.938908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.938863    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.955069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.948158    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.984462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:28.957404    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:29.037114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.005129    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:29.132125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.122310    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:29.281950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.132502    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:29.281950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.139803    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:29.287200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.287144    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:02:29.478200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.478154    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 02:02:29.486922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.486880    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:02:29.685071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.660890    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded"
Oct 31 02:02:29.893350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.888613    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:29.940567 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.940522    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:30.008746 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:29.999253    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:02:30.050066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.049390    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:02:30.159668 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.159624    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:02:30.266064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.239205    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:30.300064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.297517    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:30.300064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.298647    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:30.300064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.298785    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:30.317061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.313408    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:30.317061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.313790    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:02:30.317061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.314080    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:02:30.340879 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.340825    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:30.402098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.341156    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"]
Oct 31 02:02:30.439844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.439743    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:02:30.447773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.369561    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:30.521068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.487910    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:30.550463 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.541950    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:30.578334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.567362    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.580859 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.580832    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"payload-processing-external-model-plugins-e2e-extmod-b-df9d42\""
Oct 31 02:02:30.581230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.581214    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"routing-overlay\""
Oct 31 02:02:30.581490 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.581475    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"payload-processing-external-model-e2e--da998f83-dockercfg-5f7c7\""
Oct 31 02:02:30.581670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.581648    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"agc-cred-cfe3b0811863\""
Oct 31 02:02:30.581785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.581730    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"kube-root-ca.crt\""
Oct 31 02:02:30.581902 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.581819    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"openshift-service-ca.crt\""
Oct 31 02:02:30.591176 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.591138    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:30.591380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.591197    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:30.621092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.616328    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:30.621092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.616385    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:30.661535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.655823    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.661535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.656022    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.661535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.656130    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.661535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.656183    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh9kg\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-kube-api-access-kh9kg\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.946505 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.946461    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.946783 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.946760    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.946924 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.946911    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh9kg\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-kube-api-access-kh9kg\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.965342 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.952207    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:30.965342 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.962180    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.965701 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.965672    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:30.971481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:30.971445    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:31.045122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.041584    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh9kg\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-kube-api-access-kh9kg\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:31.047067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.046718    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:31.050502 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.049663    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:02:31.072070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.072008    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:31.072982 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.072958    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"]
Oct 31 02:02:31.073255 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.073238    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:02:31.073390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.073378    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"]
Oct 31 02:02:31.080161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.080143    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"]
Oct 31 02:02:31.080388 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.080376    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:31.084571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.084549    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-a62b50775dea\""
Oct 31 02:02:31.156149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.156111    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:31.166443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.166403    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:31.321117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.194267    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:31.380163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.380029    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-kube-api-access-sd9bm\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:31.380793 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.380769    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:31.819755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.819714    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:31.823067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.296765    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:31.827111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.384662    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:02:31.835337 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.384914    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:02:31.858945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.858897    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:31.890712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.574450    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 02:02:31.891593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.576303    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.733755    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.095229    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.733816    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.095859    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.037742    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.755939    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.097574    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.399722    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.106191    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:32.116031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.059322    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:02:32.119744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:31.475067    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:32.493782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.424245    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:32.732069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.729992    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:32.732069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.730173    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-kube-api-access-sd9bm\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:32.732069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.730420    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:32.794071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.784293    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 02:02:32.794071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.784373    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 02:02:32.794071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.784776    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:32.794071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.785686    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 31 02:02:32.794071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.785729    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 31 02:02:32.794071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.789286    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:32.826194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.815695    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:32.826663 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.822354    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:32.855702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.838723    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:02:32.856837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.856797    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:02:32.862152 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.862117    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:02:32.906444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.906395    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:32.907588 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:32.907558    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:02:33.015435 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.015399    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:33.094151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.093177    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:33.115103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.108570    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:33.115103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.114065    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:33.115103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.114450    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:02:33.115103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.114728    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:33.140724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.140562    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:02:33.166223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.149424    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"]
Oct 31 02:02:33.244545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.218393    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-kube-api-access-sd9bm\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:33.283603 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.283346    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:33.742059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.738446    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:33.814059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.810278    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 31 02:02:33.818112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.818084    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:33.818366 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.818354    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:02:33.848296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.848265    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:02:33.894667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.883997    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:02:33.894667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.856415    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:33.916483 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.915146    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:02:33.922518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.920973    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw" event={"ID":"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f","Type":"ContainerStarted","Data":"1fcbe0b7aa8d4ea9311a396beafc5b119cb2b641ebaaeeca98ee9b6542b657b3"}
Oct 31 02:02:33.924670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.923357    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:33.946232 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.946204    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:02:33.946564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.946539    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:33.962086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:33.949328    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:02:34.075935 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.074699    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:02:34.075935 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.074894    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:02:34.082276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.082188    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:02:34.116616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.116590    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:02:34.118165 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.117962    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:34.119789 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.119202    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:02:34.135349 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.135327    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:02:34.141553 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.141538    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:02:34.356301 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:02:34.356261    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dd09fc_7789_491b_bac2_c7b300b6c750.slice/crio-52daf24133463430549a0d11b5c5ebb1dcf0d7206d88f722a566e6a2cd1903c3 WatchSource:0}: Error finding container 52daf24133463430549a0d11b5c5ebb1dcf0d7206d88f722a566e6a2cd1903c3: Status 404 returned error can't find the container with id 52daf24133463430549a0d11b5c5ebb1dcf0d7206d88f722a566e6a2cd1903c3
Oct 31 02:02:34.361419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.361389    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"]
Oct 31 02:02:34.936893 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.936847    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx" event={"ID":"a3dd09fc-7789-491b-bac2-c7b300b6c750","Type":"ContainerStarted","Data":"eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674"}
Oct 31 02:02:34.937120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.936919    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx" event={"ID":"a3dd09fc-7789-491b-bac2-c7b300b6c750","Type":"ContainerStarted","Data":"52daf24133463430549a0d11b5c5ebb1dcf0d7206d88f722a566e6a2cd1903c3"}
Oct 31 02:02:34.937450 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.937426    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:34.940574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.940513    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw" event={"ID":"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f","Type":"ContainerStarted","Data":"8b054951b48170e832bd5fdea89514449cf6cca5b6ff0d338fea2c2d6c8730da"}
Oct 31 02:02:34.940849 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.940828    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:34.960680 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.960635    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx" podStartSLOduration=4.96062266 podStartE2EDuration="4.96062266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:34.959026682 +0000 UTC m=+2512.907675076" watchObservedRunningTime="2026-10-31 02:02:34.96062266 +0000 UTC m=+2512.909271074"
Oct 31 02:02:34.985514 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:34.985459    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw" podStartSLOduration=5.985440938 podStartE2EDuration="5.985440938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:34.978750952 +0000 UTC m=+2512.927399345" watchObservedRunningTime="2026-10-31 02:02:34.985440938 +0000 UTC m=+2512.934089330"
Oct 31 02:02:35.713227 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:35.713178    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:02:35.713629 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:35.713248    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:41.024525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.024486    5293 generic.go:356] "Generic (PLEG): container finished" podID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerID="b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519" exitCode=0
Oct 31 02:02:41.025129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.024688    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" event={"ID":"fc5aa93f-da9b-41e2-acc1-155a54a3767d","Type":"ContainerDied","Data":"b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519"}
Oct 31 02:02:41.025241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.025218    5293 scope.go:122] "RemoveContainer" containerID="b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519"
Oct 31 02:02:41.027815 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.027775    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:41.027941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.027869    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:41.027941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.027913    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:41.029293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.029266    5293 generic.go:356] "Generic (PLEG): container finished" podID="e8e177db-8776-47df-96ad-404ec445b4b6" containerID="5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99" exitCode=0
Oct 31 02:02:41.029389 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.029343    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" event={"ID":"e8e177db-8776-47df-96ad-404ec445b4b6","Type":"ContainerDied","Data":"5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99"}
Oct 31 02:02:41.029816 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.029794    5293 scope.go:122] "RemoveContainer" containerID="5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99"
Oct 31 02:02:41.032477 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.032447    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:41.032549 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.032532    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:41.032593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:41.032581    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:02:41.850294 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:02:41.850190    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:02:42.055127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:42.055075    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" event={"ID":"fc5aa93f-da9b-41e2-acc1-155a54a3767d","Type":"ContainerStarted","Data":"c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25"}
Oct 31 02:02:42.057903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:42.057876    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" event={"ID":"e8e177db-8776-47df-96ad-404ec445b4b6","Type":"ContainerStarted","Data":"5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a"}
Oct 31 02:02:42.146453 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:42.146415    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:02:42.146674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:42.146543    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 31 02:02:42.146674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:42.146585    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 31 02:02:42.284938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:42.284908    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:02:42.285155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:42.285136    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 31 02:02:42.285209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:42.285177    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 31 02:02:43.157111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:43.138385    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 31 02:02:43.157111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:43.138466    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 31 02:02:43.286500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:43.286454    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 31 02:02:43.286816 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:43.286766    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 31 02:02:44.131063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:44.117359    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 31 02:02:44.131063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:44.117420    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 31 02:02:44.285697 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:44.285669    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 31 02:02:44.286139 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:44.285720    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 31 02:02:45.119137 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.119052    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 31 02:02:45.119481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.119435    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 31 02:02:45.294085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.293732    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 31 02:02:45.294085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.293790    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 31 02:02:45.870157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.870123    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"]
Oct 31 02:02:45.883236 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.882554    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:45.885961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.885216    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"]
Oct 31 02:02:45.888560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.888022    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c088fd-serving-cert\""
Oct 31 02:02:45.888560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.888444    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjq8v\""
Oct 31 02:02:45.888776 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.888593    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:02:45.888776 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.888678    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:02:45.937825 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.937771    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 02:02:45.966535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.965110    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:02:45.968632 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.967718    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:02:45.982650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.982604    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx5vk\" (UniqueName: \"kubernetes.io/projected/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-kube-api-access-sx5vk\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:45.982852 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.982838    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:45.983056 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.983026    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:45.983262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:45.983251    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-metrics-certs\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.085899 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.085855    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-metrics-certs\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.086277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.086253    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx5vk\" (UniqueName: \"kubernetes.io/projected/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-kube-api-access-sx5vk\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.087552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.086763    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.087552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.086924    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.087552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.087500    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.090966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.090945    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.091897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.091870    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-metrics-certs\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.093687 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.093668    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"]
Oct 31 02:02:46.199088 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.178092    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 31 02:02:46.199088 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.178153    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 31 02:02:46.422075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.403713    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx5vk\" (UniqueName: \"kubernetes.io/projected/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-kube-api-access-sx5vk\") pod \"maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.518118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.506799    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:46.902127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.845025    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 31 02:02:46.902127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:46.845112    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 31 02:02:47.210117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:47.206580    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:02:47.210117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:47.208474    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 31 02:02:47.210117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:47.208534    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 31 02:02:47.289239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:47.289193    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:47.322503 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:47.322452    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 31 02:02:47.322828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:47.322785    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 31 02:02:48.079148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:48.079064    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"]
Oct 31 02:02:48.079718 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:02:48.079691    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cf765a_5a4f_4f39_828e_0aff2dcb96dc.slice/crio-49fbdb1c48f86d520aff544df628a0a53a429a63ac9b1e71c1e6ed7d05f9f52b WatchSource:0}: Error finding container 49fbdb1c48f86d520aff544df628a0a53a429a63ac9b1e71c1e6ed7d05f9f52b: Status 404 returned error can't find the container with id 49fbdb1c48f86d520aff544df628a0a53a429a63ac9b1e71c1e6ed7d05f9f52b
Oct 31 02:02:48.083197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:48.083119    5293 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:02:48.120027 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:48.119991    5293 patch_prober.go:29] interesting pod/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 31 02:02:48.120305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:48.120276    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 31 02:02:48.285363 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:48.285320    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 31 02:02:48.285543 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:48.285378    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 31 02:02:48.659434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:48.658155    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:02:48.896931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:48.896899    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq" event={"ID":"91cf765a-5a4f-4f39-828e-0aff2dcb96dc","Type":"ContainerStarted","Data":"49fbdb1c48f86d520aff544df628a0a53a429a63ac9b1e71c1e6ed7d05f9f52b"}
Oct 31 02:02:49.142197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:49.141600    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:02:49.145432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:49.145024    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:02:49.301957 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:49.301848    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:02:49.304417 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:49.303583    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:02:49.316094 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:49.305833    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:02:51.106065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:51.105391    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht" podUID="2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" containerName="payload-processing" containerID="cri-o://6110044fb2ef081435b22c24e4bd4bb259de99e72fa912123f93f6e99a59a64c" gracePeriod=25
Oct 31 02:02:52.236320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.236237    5293 generic.go:356] "Generic (PLEG): container finished" podID="2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" containerID="6110044fb2ef081435b22c24e4bd4bb259de99e72fa912123f93f6e99a59a64c" exitCode=0
Oct 31 02:02:52.237026 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.237003    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht" event={"ID":"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d","Type":"ContainerDied","Data":"6110044fb2ef081435b22c24e4bd4bb259de99e72fa912123f93f6e99a59a64c"}
Oct 31 02:02:52.237205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.237188    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht" event={"ID":"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d","Type":"ContainerDied","Data":"ea3de0d1dc9601331ad27684318c25989682daaaadf1753b40e6ca9aab3ffd33"}
Oct 31 02:02:52.237305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.237294    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3de0d1dc9601331ad27684318c25989682daaaadf1753b40e6ca9aab3ffd33"
Oct 31 02:02:52.243256 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.243195    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:02:52.406074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.404907    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-kube-api-access-xrbbl\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-kube-api-access-xrbbl\") pod \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") "
Oct 31 02:02:52.406074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.405055    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-plugins-config-volume\") pod \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") "
Oct 31 02:02:52.406074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.405344    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-routing-overlay-volume\") pod \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") "
Oct 31 02:02:52.406074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.405514    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-provider-credentials\") pod \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\" (UID: \"2c5d11c6-c841-4213-a8ca-ba6ddb76be3d\") "
Oct 31 02:02:52.468578 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.465251    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-kube-api-access-xrbbl" pod "2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" (UID: "2c5d11c6-c841-4213-a8ca-ba6ddb76be3d"). InnerVolumeSpecName "kube-api-access-xrbbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:02:52.468578 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.466100    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-provider-credentials" pod "2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" (UID: "2c5d11c6-c841-4213-a8ca-ba6ddb76be3d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:02:52.469476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.468997    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-routing-overlay-volume" pod "2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" (UID: "2c5d11c6-c841-4213-a8ca-ba6ddb76be3d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:02:52.478304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.477420    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-plugins-config-volume" pod "2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" (UID: "2c5d11c6-c841-4213-a8ca-ba6ddb76be3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:02:52.511385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.510262    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:52.511385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.510293    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:52.511385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.510303    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrbbl\" (UniqueName: \"kubernetes.io/projected/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-kube-api-access-xrbbl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:52.511385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:52.510314    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:53.268070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:53.266568    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"
Oct 31 02:02:53.301789 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:53.301763    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"]
Oct 31 02:02:53.306780 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:53.306757    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7756ht"]
Oct 31 02:02:54.226695 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:54.226654    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" path="/var/lib/kubelet/pods/2c5d11c6-c841-4213-a8ca-ba6ddb76be3d/volumes"
Oct 31 02:02:54.278330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:54.278283    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq" event={"ID":"91cf765a-5a4f-4f39-828e-0aff2dcb96dc","Type":"ContainerStarted","Data":"3f9887dc15addf51c66641c64c8d802d6ef9febe05b3645c2d3a5ce0bf5a1e2b"}
Oct 31 02:02:54.278800 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:54.278369    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:02:54.302828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:54.302770    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq" podStartSLOduration=4.09797878 podStartE2EDuration="9.302753102s" totalImagesPullingTime="5.204774322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:54.297661398 +0000 UTC m=+2532.246309802" watchObservedRunningTime="2026-10-31 02:02:54.302753102 +0000 UTC m=+2532.251401496"
Oct 31 02:02:55.109066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.055610    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"]
Oct 31 02:02:55.109066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.056986    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" containerName="payload-processing"
Oct 31 02:02:55.109066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.057006    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" containerName="payload-processing"
Oct 31 02:02:55.109066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.057222    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5d11c6-c841-4213-a8ca-ba6ddb76be3d" containerName="payload-processing"
Oct 31 02:02:55.109066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.066330    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.109066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.108549    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-28af2be07c61\""
Oct 31 02:02:55.128977 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.128944    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"]
Oct 31 02:02:55.228308 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.228272    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.228514 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.228406    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.228514 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.228504    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.228658 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.228637    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2r2d\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-kube-api-access-s2r2d\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.330360 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.330318    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.330892 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.330415    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.330892 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.330536    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.330892 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.330607    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r2d\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-kube-api-access-s2r2d\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.331166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.331146    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.331374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.331353    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.333514 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.333452    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.343251 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.343219    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2r2d\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-kube-api-access-s2r2d\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.433423 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.433376    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:55.788470 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:02:55.770952    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2fd35d_14be_48ba_b657_9752a72429b2.slice/crio-ae5b500a5584af556c653b7233f90a2457306b59259e9aa4d0f672b1681d84cb WatchSource:0}: Error finding container ae5b500a5584af556c653b7233f90a2457306b59259e9aa4d0f672b1681d84cb: Status 404 returned error can't find the container with id ae5b500a5584af556c653b7233f90a2457306b59259e9aa4d0f672b1681d84cb
Oct 31 02:02:55.788470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:55.776907    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"]
Oct 31 02:02:56.505535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.505495    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" event={"ID":"2e2fd35d-14be-48ba-b657-9752a72429b2","Type":"ContainerStarted","Data":"ae5b500a5584af556c653b7233f90a2457306b59259e9aa4d0f672b1681d84cb"}
Oct 31 02:02:56.537857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.523722    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"]
Oct 31 02:02:56.537857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.537088    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"]
Oct 31 02:02:56.537857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.537224    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.543212 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.541573    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"agc-cred-28829414036f\""
Oct 31 02:02:56.543212 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.541660    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"agc-cred-c2cf8b86fc75\""
Oct 31 02:02:56.545897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.545685    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"kube-root-ca.crt\""
Oct 31 02:02:56.545897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.545714    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"openshift-service-ca.crt\""
Oct 31 02:02:56.545897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.545755    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"routing-overlay\""
Oct 31 02:02:56.545897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.545787    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff\""
Oct 31 02:02:56.545897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.545819    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"payload-processing-external-model-e2e--4f663f8f-dockercfg-k6x5j\""
Oct 31 02:02:56.608874 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.608827    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.609107 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.608967    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.609107 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.609025    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56b6w\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-kube-api-access-56b6w\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.609262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.609165    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.711026 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.710982    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56b6w\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-kube-api-access-56b6w\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.711213 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.711107    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.711559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.711535    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.711686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.711608    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.712221 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.712195    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.712688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.712662    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.714239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.714215    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.723807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.723780    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56b6w\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-kube-api-access-56b6w\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:56.867097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:56.866977    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:57.172871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:57.172835    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"]
Oct 31 02:02:57.182975 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:02:57.182948    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1282bdd_59fa_4e77_87cc_84182fd71d46.slice/crio-4c4851aa09e3f00b40d68fc76d303b98796793e012f5d6c154ac005c6eaeba95 WatchSource:0}: Error finding container 4c4851aa09e3f00b40d68fc76d303b98796793e012f5d6c154ac005c6eaeba95: Status 404 returned error can't find the container with id 4c4851aa09e3f00b40d68fc76d303b98796793e012f5d6c154ac005c6eaeba95
Oct 31 02:02:57.515593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:57.515498    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" event={"ID":"2e2fd35d-14be-48ba-b657-9752a72429b2","Type":"ContainerStarted","Data":"a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0"}
Oct 31 02:02:57.516153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:57.515615    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:02:57.517915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:57.517889    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz" event={"ID":"d1282bdd-59fa-4e77-87cc-84182fd71d46","Type":"ContainerStarted","Data":"de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73"}
Oct 31 02:02:57.518019 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:57.517936    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz" event={"ID":"d1282bdd-59fa-4e77-87cc-84182fd71d46","Type":"ContainerStarted","Data":"4c4851aa09e3f00b40d68fc76d303b98796793e012f5d6c154ac005c6eaeba95"}
Oct 31 02:02:57.518101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:57.518018    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:02:57.537654 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:57.537615    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" podStartSLOduration=3.537603132 podStartE2EDuration="3.537603132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:57.53299112 +0000 UTC m=+2535.481639540" watchObservedRunningTime="2026-10-31 02:02:57.537603132 +0000 UTC m=+2535.486251523"
Oct 31 02:02:57.552377 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:02:57.552336    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz" podStartSLOduration=1.55232412 podStartE2EDuration="1.55232412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:57.550325639 +0000 UTC m=+2535.498974031" watchObservedRunningTime="2026-10-31 02:02:57.55232412 +0000 UTC m=+2535.500972539"
Oct 31 02:03:00.292181 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:00.292151    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:03:08.616289 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:08.616255    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:03:08.631549 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:08.626254    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:03:08.760932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:08.760894    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"]
Oct 31 02:03:11.784546 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:03:11.784508    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:03:13.769311 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:13.768327    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx" podUID="a3dd09fc-7789-491b-bac2-c7b300b6c750" containerName="payload-processing" containerID="cri-o://eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674" gracePeriod=25
Oct 31 02:03:14.284032 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.284006    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:03:14.321806 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.321776    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-routing-overlay-volume\") pod \"a3dd09fc-7789-491b-bac2-c7b300b6c750\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") "
Oct 31 02:03:14.321999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.321844    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-plugins-config-volume\") pod \"a3dd09fc-7789-491b-bac2-c7b300b6c750\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") "
Oct 31 02:03:14.322187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.322034    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-kube-api-access-sd9bm\") pod \"a3dd09fc-7789-491b-bac2-c7b300b6c750\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") "
Oct 31 02:03:14.322187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.322132    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-provider-credentials\") pod \"a3dd09fc-7789-491b-bac2-c7b300b6c750\" (UID: \"a3dd09fc-7789-491b-bac2-c7b300b6c750\") "
Oct 31 02:03:14.322769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.322744    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-plugins-config-volume" pod "a3dd09fc-7789-491b-bac2-c7b300b6c750" (UID: "a3dd09fc-7789-491b-bac2-c7b300b6c750"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:03:14.323201 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.323180    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:03:14.323477 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.323450    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-routing-overlay-volume" pod "a3dd09fc-7789-491b-bac2-c7b300b6c750" (UID: "a3dd09fc-7789-491b-bac2-c7b300b6c750"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:03:14.326422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.326394    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-provider-credentials" pod "a3dd09fc-7789-491b-bac2-c7b300b6c750" (UID: "a3dd09fc-7789-491b-bac2-c7b300b6c750"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:03:14.336568 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.336541    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-kube-api-access-sd9bm" pod "a3dd09fc-7789-491b-bac2-c7b300b6c750" (UID: "a3dd09fc-7789-491b-bac2-c7b300b6c750"). InnerVolumeSpecName "kube-api-access-sd9bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:03:14.433426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.433395    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-kube-api-access-sd9bm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:03:14.433426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.433424    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3dd09fc-7789-491b-bac2-c7b300b6c750-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:03:14.433651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.433434    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3dd09fc-7789-491b-bac2-c7b300b6c750-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:03:14.887145 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.887104    5293 generic.go:356] "Generic (PLEG): container finished" podID="a3dd09fc-7789-491b-bac2-c7b300b6c750" containerID="eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674" exitCode=0
Oct 31 02:03:14.887950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.887914    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx" event={"ID":"a3dd09fc-7789-491b-bac2-c7b300b6c750","Type":"ContainerDied","Data":"eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674"}
Oct 31 02:03:14.888158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.888031    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx" event={"ID":"a3dd09fc-7789-491b-bac2-c7b300b6c750","Type":"ContainerDied","Data":"52daf24133463430549a0d11b5c5ebb1dcf0d7206d88f722a566e6a2cd1903c3"}
Oct 31 02:03:14.888158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.888100    5293 scope.go:122] "RemoveContainer" containerID="eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674"
Oct 31 02:03:14.888332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.888313    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"
Oct 31 02:03:14.906242 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.906220    5293 scope.go:122] "RemoveContainer" containerID="eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674"
Oct 31 02:03:14.906535 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:03:14.906515    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674\": container with ID starting with eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674 not found: ID does not exist" containerID="eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674"
Oct 31 02:03:14.906609 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.906544    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674"} err="failed to get container status \"eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674\": rpc error: code = NotFound desc = could not find container \"eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674\": container with ID starting with eacc0a9e4d87713d9d8671d9e2aa77b0bc835855cd0febfb723841fb5f230674 not found: ID does not exist"
Oct 31 02:03:14.921979 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.921952    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"]
Oct 31 02:03:14.926565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:14.926535    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6789qj5jx"]
Oct 31 02:03:16.209056 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:16.208990    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3dd09fc-7789-491b-bac2-c7b300b6c750" path="/var/lib/kubelet/pods/a3dd09fc-7789-491b-bac2-c7b300b6c750/volumes"
Oct 31 02:03:25.184463 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:25.184432    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:03:41.881275 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:03:41.881236    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:03:47.403626 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:47.403596    5293 scope.go:122] "RemoveContainer" containerID="874f6cabacb88195dfa54a1920fc866e9458e2f428b5cc152d3ac0a505c2c1dd"
Oct 31 02:03:51.554380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:51.554304    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:03:51.910491 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:51.910445    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:03:56.349294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.349238    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"]
Oct 31 02:03:56.350408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.350385    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a3dd09fc-7789-491b-bac2-c7b300b6c750" containerName="payload-processing"
Oct 31 02:03:56.350408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.350406    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dd09fc-7789-491b-bac2-c7b300b6c750" containerName="payload-processing"
Oct 31 02:03:56.350561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.350550    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dd09fc-7789-491b-bac2-c7b300b6c750" containerName="payload-processing"
Oct 31 02:03:56.356320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.356296    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.360999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.360962    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"]
Oct 31 02:03:56.361310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.361235    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"agc-cred-2fedf77eafb9\""
Oct 31 02:03:56.395687 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.395653    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bmxw\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-kube-api-access-9bmxw\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.395850 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.395762    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.396065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.396025    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.396194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.396148    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.497499 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.497459    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bmxw\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-kube-api-access-9bmxw\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.497703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.497521    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.497785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.497765    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.497870 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.497846    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.498440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.498400    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.498612 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.498586    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.500633 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.500611    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.509494 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.509472    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bmxw\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-kube-api-access-9bmxw\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:56.685939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:56.685908    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:57.046756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:57.046610    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"]
Oct 31 02:03:57.058851 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:03:57.058819    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091dea8e_578a_4fb6_9fa1_16302e983ac5.slice/crio-0957ca371e06b45a89c589259e7cdfd3454375991151599b12714de09cdcb949 WatchSource:0}: Error finding container 0957ca371e06b45a89c589259e7cdfd3454375991151599b12714de09cdcb949: Status 404 returned error can't find the container with id 0957ca371e06b45a89c589259e7cdfd3454375991151599b12714de09cdcb949
Oct 31 02:03:57.895330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:57.895280    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs" event={"ID":"091dea8e-578a-4fb6-9fa1-16302e983ac5","Type":"ContainerStarted","Data":"05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe"}
Oct 31 02:03:57.896286 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:57.896185    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:03:57.896286 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:57.896213    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs" event={"ID":"091dea8e-578a-4fb6-9fa1-16302e983ac5","Type":"ContainerStarted","Data":"0957ca371e06b45a89c589259e7cdfd3454375991151599b12714de09cdcb949"}
Oct 31 02:03:57.918513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:03:57.918469    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs" podStartSLOduration=1.918455699 podStartE2EDuration="1.918455699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:03:57.914896821 +0000 UTC m=+2595.863545213" watchObservedRunningTime="2026-10-31 02:03:57.918455699 +0000 UTC m=+2595.867104113"
Oct 31 02:04:08.998376 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:08.998348    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:04:09.056441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:09.056409    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"]
Oct 31 02:04:09.361839 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:09.361769    5293 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0b6abf/agc-cred-28af2be07c61: secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:09.362073 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:09.362056    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778: secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:09.362313 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:09.362283    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials podName:2e2fd35d-14be-48ba-b657-9752a72429b2 nodeName:}" failed. No retries permitted until 2026-10-31 02:04:09.86225934 +0000 UTC m=+2607.810907730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" (UID: "2e2fd35d-14be-48ba-b657-9752a72429b2") : secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:09.886068 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:09.875882    5293 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0b6abf/agc-cred-28af2be07c61: secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:09.886068 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:09.875919    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778: secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:09.886068 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:09.876014    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials podName:2e2fd35d-14be-48ba-b657-9752a72429b2 nodeName:}" failed. No retries permitted until 2026-10-31 02:04:10.875993127 +0000 UTC m=+2608.824641514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" (UID: "2e2fd35d-14be-48ba-b657-9752a72429b2") : secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:10.885951 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:10.885906    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"]
Oct 31 02:04:10.904234 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:10.904195    5293 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0b6abf/agc-cred-28af2be07c61: secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:10.904484 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:10.904465    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778: secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:10.904679 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:10.904666    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials podName:2e2fd35d-14be-48ba-b657-9752a72429b2 nodeName:}" failed. No retries permitted until 2026-10-31 02:04:12.904643735 +0000 UTC m=+2610.853292125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" (UID: "2e2fd35d-14be-48ba-b657-9752a72429b2") : secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:11.783269 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:11.783230    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:04:12.974292 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:12.974254    5293 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0b6abf/agc-cred-28af2be07c61: secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:12.974292 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:12.974282    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778: secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:12.974873 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:12.974335    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials podName:2e2fd35d-14be-48ba-b657-9752a72429b2 nodeName:}" failed. No retries permitted until 2026-10-31 02:04:16.97432206 +0000 UTC m=+2614.922970442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" (UID: "2e2fd35d-14be-48ba-b657-9752a72429b2") : secret "agc-cred-28af2be07c61" not found
Oct 31 02:04:14.063196 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.063144    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j" podUID="bf575678-3aed-403c-8a18-8e94699edbac" containerName="payload-processing" containerID="cri-o://e5966c3487a3e38cd47288e2f0d62349baa2724ee5b220ed537e4506fb1f1ae4" gracePeriod=25
Oct 31 02:04:14.326777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.326693    5293 generic.go:356] "Generic (PLEG): container finished" podID="bf575678-3aed-403c-8a18-8e94699edbac" containerID="e5966c3487a3e38cd47288e2f0d62349baa2724ee5b220ed537e4506fb1f1ae4" exitCode=0
Oct 31 02:04:14.326932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.326769    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j" event={"ID":"bf575678-3aed-403c-8a18-8e94699edbac","Type":"ContainerDied","Data":"e5966c3487a3e38cd47288e2f0d62349baa2724ee5b220ed537e4506fb1f1ae4"}
Oct 31 02:04:14.405174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.405142    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:04:14.510339 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.510304    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-provider-credentials\") pod \"bf575678-3aed-403c-8a18-8e94699edbac\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") "
Oct 31 02:04:14.510534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.510425    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-kube-api-access-w5sdf\") pod \"bf575678-3aed-403c-8a18-8e94699edbac\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") "
Oct 31 02:04:14.510604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.510572    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-routing-overlay-volume\") pod \"bf575678-3aed-403c-8a18-8e94699edbac\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") "
Oct 31 02:04:14.510812 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.510791    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-plugins-config-volume\") pod \"bf575678-3aed-403c-8a18-8e94699edbac\" (UID: \"bf575678-3aed-403c-8a18-8e94699edbac\") "
Oct 31 02:04:14.511236 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.511201    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-plugins-config-volume" pod "bf575678-3aed-403c-8a18-8e94699edbac" (UID: "bf575678-3aed-403c-8a18-8e94699edbac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:14.511400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.511377    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-routing-overlay-volume" pod "bf575678-3aed-403c-8a18-8e94699edbac" (UID: "bf575678-3aed-403c-8a18-8e94699edbac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:14.512067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.512030    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:14.512199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.512184    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf575678-3aed-403c-8a18-8e94699edbac-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:14.513106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.513027    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-provider-credentials" pod "bf575678-3aed-403c-8a18-8e94699edbac" (UID: "bf575678-3aed-403c-8a18-8e94699edbac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:14.513419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.513399    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-kube-api-access-w5sdf" pod "bf575678-3aed-403c-8a18-8e94699edbac" (UID: "bf575678-3aed-403c-8a18-8e94699edbac"). InnerVolumeSpecName "kube-api-access-w5sdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:14.615085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.614938    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:14.615085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:14.615002    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/bf575678-3aed-403c-8a18-8e94699edbac-kube-api-access-w5sdf\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:15.432557 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:15.432445    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j" event={"ID":"bf575678-3aed-403c-8a18-8e94699edbac","Type":"ContainerDied","Data":"3b73e3e174abefd1d68fe357d035794abcf028d3d1823e50c91e7a8133ca9e57"}
Oct 31 02:04:15.433205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:15.433182    5293 scope.go:122] "RemoveContainer" containerID="e5966c3487a3e38cd47288e2f0d62349baa2724ee5b220ed537e4506fb1f1ae4"
Oct 31 02:04:15.433470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:15.432575    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"
Oct 31 02:04:15.465966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:15.465920    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"]
Oct 31 02:04:15.470973 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:15.470946    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-7c66hxl6j"]
Oct 31 02:04:15.886649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:15.886550    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" podUID="2e2fd35d-14be-48ba-b657-9752a72429b2" containerName="payload-processing" containerID="cri-o://a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0" gracePeriod=25
Oct 31 02:04:16.176893 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.176858    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf575678-3aed-403c-8a18-8e94699edbac" path="/var/lib/kubelet/pods/bf575678-3aed-403c-8a18-8e94699edbac/volumes"
Oct 31 02:04:16.311537 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.311510    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:04:16.348101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.347593    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"]
Oct 31 02:04:16.348670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.348649    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2e2fd35d-14be-48ba-b657-9752a72429b2" containerName="payload-processing"
Oct 31 02:04:16.348807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.348672    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2fd35d-14be-48ba-b657-9752a72429b2" containerName="payload-processing"
Oct 31 02:04:16.348807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.348716    5293 cpu_manager.go:414] "containerMap: removing container" podUID="bf575678-3aed-403c-8a18-8e94699edbac" containerName="payload-processing"
Oct 31 02:04:16.348807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.348724    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf575678-3aed-403c-8a18-8e94699edbac" containerName="payload-processing"
Oct 31 02:04:16.348964 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.348947    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2fd35d-14be-48ba-b657-9752a72429b2" containerName="payload-processing"
Oct 31 02:04:16.349019 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.348968    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf575678-3aed-403c-8a18-8e94699edbac" containerName="payload-processing"
Oct 31 02:04:16.350869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.350839    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials\") pod \"2e2fd35d-14be-48ba-b657-9752a72429b2\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") "
Oct 31 02:04:16.351024 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.350931    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-plugins-config-volume\") pod \"2e2fd35d-14be-48ba-b657-9752a72429b2\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") "
Oct 31 02:04:16.351024 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.350986    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-routing-overlay-volume\") pod \"2e2fd35d-14be-48ba-b657-9752a72429b2\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") "
Oct 31 02:04:16.351206 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.351183    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-kube-api-access-s2r2d\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-kube-api-access-s2r2d\") pod \"2e2fd35d-14be-48ba-b657-9752a72429b2\" (UID: \"2e2fd35d-14be-48ba-b657-9752a72429b2\") "
Oct 31 02:04:16.352441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.352406    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-routing-overlay-volume" pod "2e2fd35d-14be-48ba-b657-9752a72429b2" (UID: "2e2fd35d-14be-48ba-b657-9752a72429b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:16.352866 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.352617    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-plugins-config-volume" pod "2e2fd35d-14be-48ba-b657-9752a72429b2" (UID: "2e2fd35d-14be-48ba-b657-9752a72429b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:16.355932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.354147    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.355932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.354572    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials" pod "2e2fd35d-14be-48ba-b657-9752a72429b2" (UID: "2e2fd35d-14be-48ba-b657-9752a72429b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:16.369972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.366313    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"]
Oct 31 02:04:16.369972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.368678    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c088fd-openshift-default-dockercfg-tdwrg\""
Oct 31 02:04:16.381657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.381610    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-kube-api-access-s2r2d" pod "2e2fd35d-14be-48ba-b657-9752a72429b2" (UID: "2e2fd35d-14be-48ba-b657-9752a72429b2"). InnerVolumeSpecName "kube-api-access-s2r2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:16.454638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.454475    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rd5\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-kube-api-access-x2rd5\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.455254 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.455226    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.455553 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.455522    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-token\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.455732 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.455712    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-certs\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.455865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.455786    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-podinfo\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.455865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.455840    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istiod-ca-cert\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.456005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.455913    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-credential-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.456005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.455951    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-data\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.456005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.455988    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-envoy\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.456330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.456292    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:16.456330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.456317    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:16.456482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.456332    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e2fd35d-14be-48ba-b657-9752a72429b2-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:16.456482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.456370    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2r2d\" (UniqueName: \"kubernetes.io/projected/2e2fd35d-14be-48ba-b657-9752a72429b2-kube-api-access-s2r2d\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:16.465565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.465531    5293 generic.go:356] "Generic (PLEG): container finished" podID="2e2fd35d-14be-48ba-b657-9752a72429b2" containerID="a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0" exitCode=0
Oct 31 02:04:16.465768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.465749    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" event={"ID":"2e2fd35d-14be-48ba-b657-9752a72429b2","Type":"ContainerDied","Data":"a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0"}
Oct 31 02:04:16.465849 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.465809    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778" event={"ID":"2e2fd35d-14be-48ba-b657-9752a72429b2","Type":"ContainerDied","Data":"ae5b500a5584af556c653b7233f90a2457306b59259e9aa4d0f672b1681d84cb"}
Oct 31 02:04:16.465849 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.465842    5293 scope.go:122] "RemoveContainer" containerID="a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0"
Oct 31 02:04:16.465995 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.465981    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"
Oct 31 02:04:16.478963 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.478945    5293 scope.go:122] "RemoveContainer" containerID="a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0"
Oct 31 02:04:16.479278 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:16.479256    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0\": container with ID starting with a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0 not found: ID does not exist" containerID="a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0"
Oct 31 02:04:16.479331 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.479291    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0"} err="failed to get container status \"a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0\": rpc error: code = NotFound desc = could not find container \"a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0\": container with ID starting with a176e17e52a39ec5e2fbf1da39877d9861898ad3da98121aae20acdb9209a3c0 not found: ID does not exist"
Oct 31 02:04:16.494646 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.494622    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"]
Oct 31 02:04:16.499570 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.499547    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-58c8dw778"]
Oct 31 02:04:16.557426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557385    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rd5\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-kube-api-access-x2rd5\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.557627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557441    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.557627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557490    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-token\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.557627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557583    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-certs\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.557830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557642    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-podinfo\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.557830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557700    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istiod-ca-cert\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.557830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557781    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-credential-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.557830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557815    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-data\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.558091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557866    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-envoy\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.558091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557906    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.558091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.557946    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-certs\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.558484 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.558453    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-data\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.558615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.558510    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-credential-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.558615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.558580    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istiod-ca-cert\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.560285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.560263    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-envoy\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.561258 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.561238    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-podinfo\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.569223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.569202    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rd5\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-kube-api-access-x2rd5\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.569320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.569237    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-token\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:16.721118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:16.719461    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:17.108320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:17.102681    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"]
Oct 31 02:04:17.109659 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:04:17.109614    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb569eb_af9a_4edf_aa91_a8296743a9b1.slice/crio-1fe0ff84f174b698e99abb6e1828edda94bc0c8079312abff5a4bb4e2a9d8eaa WatchSource:0}: Error finding container 1fe0ff84f174b698e99abb6e1828edda94bc0c8079312abff5a4bb4e2a9d8eaa: Status 404 returned error can't find the container with id 1fe0ff84f174b698e99abb6e1828edda94bc0c8079312abff5a4bb4e2a9d8eaa
Oct 31 02:04:17.119105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:17.119070    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:04:17.119223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:17.119188    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:04:17.119292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:17.119235    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:04:17.540020 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:17.539976    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" event={"ID":"4cb569eb-af9a-4edf-aa91-a8296743a9b1","Type":"ContainerStarted","Data":"d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061"}
Oct 31 02:04:17.548187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:17.548147    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" event={"ID":"4cb569eb-af9a-4edf-aa91-a8296743a9b1","Type":"ContainerStarted","Data":"1fe0ff84f174b698e99abb6e1828edda94bc0c8079312abff5a4bb4e2a9d8eaa"}
Oct 31 02:04:17.720363 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:17.720320    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:18.247063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:18.246836    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2fd35d-14be-48ba-b657-9752a72429b2" path="/var/lib/kubelet/pods/2e2fd35d-14be-48ba-b657-9752a72429b2/volumes"
Oct 31 02:04:18.737067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:18.734823    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Oct 31 02:04:18.737067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:18.734889    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Oct 31 02:04:19.731868 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:19.730165    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Oct 31 02:04:19.731868 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:19.730245    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Oct 31 02:04:20.722066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:20.721124    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Oct 31 02:04:20.722066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:20.721183    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Oct 31 02:04:21.722489 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:21.722450    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Oct 31 02:04:21.723110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:21.723076    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Oct 31 02:04:22.723065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:22.722419    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Oct 31 02:04:22.723065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:22.722481    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Oct 31 02:04:23.723062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:23.722402    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Oct 31 02:04:23.723062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:23.722464    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Oct 31 02:04:24.993064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:24.986619    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:25.006630 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:25.006531    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:25.056487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:25.056403    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" podStartSLOduration=37.056340427 podStartE2EDuration="37.056340427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:17.574154016 +0000 UTC m=+2615.522802408" watchObservedRunningTime="2026-10-31 02:04:25.056340427 +0000 UTC m=+2623.004988819"
Oct 31 02:04:25.061174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:25.061115    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:04:27.653102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:27.653022    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:04:30.830782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.830750    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"]
Oct 31 02:04:30.837479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.837454    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.841888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.841848    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:04:30.842321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.842297    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:04:30.849883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.849847    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"]
Oct 31 02:04:30.858636 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.858604    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5n2\" (UniqueName: \"kubernetes.io/projected/ad39557d-531b-486d-90f8-c63f8b8e12fa-kube-api-access-mb5n2\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.858775 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.858745    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.858963 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.858942    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.960392 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.960351    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5n2\" (UniqueName: \"kubernetes.io/projected/ad39557d-531b-486d-90f8-c63f8b8e12fa-kube-api-access-mb5n2\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.960586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.960459    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.960664 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.960644    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.961493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.961470    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.963098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.963077    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:30.971958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:30.971934    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5n2\" (UniqueName: \"kubernetes.io/projected/ad39557d-531b-486d-90f8-c63f8b8e12fa-kube-api-access-mb5n2\") pod \"maas-api-revoke-sub-bafdfd2f-5ms8f\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:31.173353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:31.172166    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:32.834798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:32.813252    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"]
Oct 31 02:04:32.864013 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:04:32.863679    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad39557d_531b_486d_90f8_c63f8b8e12fa.slice/crio-8f56b15e85bb8a6f68cbae68e3ff09e414edec394e5e3a10b0ad00f3d3db20dc WatchSource:0}: Error finding container 8f56b15e85bb8a6f68cbae68e3ff09e414edec394e5e3a10b0ad00f3d3db20dc: Status 404 returned error can't find the container with id 8f56b15e85bb8a6f68cbae68e3ff09e414edec394e5e3a10b0ad00f3d3db20dc
Oct 31 02:04:32.933430 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:32.925822    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:04:33.474073 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:33.473078    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f" event={"ID":"ad39557d-531b-486d-90f8-c63f8b8e12fa","Type":"ContainerStarted","Data":"8f56b15e85bb8a6f68cbae68e3ff09e414edec394e5e3a10b0ad00f3d3db20dc"}
Oct 31 02:04:34.607178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:34.607143    5293 generic.go:356] "Generic (PLEG): container finished" podID="ad39557d-531b-486d-90f8-c63f8b8e12fa" containerID="456d57683fd96c2b2762c0e43cc655a6470d86d80069fc55b242e3fd83a03fb7" exitCode=0
Oct 31 02:04:34.608001 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:34.607971    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f" event={"ID":"ad39557d-531b-486d-90f8-c63f8b8e12fa","Type":"ContainerDied","Data":"456d57683fd96c2b2762c0e43cc655a6470d86d80069fc55b242e3fd83a03fb7"}
Oct 31 02:04:36.598160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.598027    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:36.635702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.635665    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"
Oct 31 02:04:36.635882 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.635700    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f" event={"ID":"ad39557d-531b-486d-90f8-c63f8b8e12fa","Type":"ContainerDied","Data":"8f56b15e85bb8a6f68cbae68e3ff09e414edec394e5e3a10b0ad00f3d3db20dc"}
Oct 31 02:04:36.635882 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.635762    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f56b15e85bb8a6f68cbae68e3ff09e414edec394e5e3a10b0ad00f3d3db20dc"
Oct 31 02:04:36.694550 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.694514    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-xks-ca\") pod \"ad39557d-531b-486d-90f8-c63f8b8e12fa\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") "
Oct 31 02:04:36.694755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.694630    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-openshift-ca\") pod \"ad39557d-531b-486d-90f8-c63f8b8e12fa\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") "
Oct 31 02:04:36.694861 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.694836    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad39557d-531b-486d-90f8-c63f8b8e12fa-kube-api-access-mb5n2\" (UniqueName: \"kubernetes.io/projected/ad39557d-531b-486d-90f8-c63f8b8e12fa-kube-api-access-mb5n2\") pod \"ad39557d-531b-486d-90f8-c63f8b8e12fa\" (UID: \"ad39557d-531b-486d-90f8-c63f8b8e12fa\") "
Oct 31 02:04:36.694953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.694931    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-openshift-ca" pod "ad39557d-531b-486d-90f8-c63f8b8e12fa" (UID: "ad39557d-531b-486d-90f8-c63f8b8e12fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:36.695622 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.695598    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:36.697174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.697151    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad39557d-531b-486d-90f8-c63f8b8e12fa-kube-api-access-mb5n2" pod "ad39557d-531b-486d-90f8-c63f8b8e12fa" (UID: "ad39557d-531b-486d-90f8-c63f8b8e12fa"). InnerVolumeSpecName "kube-api-access-mb5n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:36.697406 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.697385    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-xks-ca" pod "ad39557d-531b-486d-90f8-c63f8b8e12fa" (UID: "ad39557d-531b-486d-90f8-c63f8b8e12fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:36.797353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.797313    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb5n2\" (UniqueName: \"kubernetes.io/projected/ad39557d-531b-486d-90f8-c63f8b8e12fa-kube-api-access-mb5n2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:36.797353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:36.797342    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad39557d-531b-486d-90f8-c63f8b8e12fa-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:41.971067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:41.967716    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"]
Oct 31 02:04:41.971067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:41.969253    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ad39557d-531b-486d-90f8-c63f8b8e12fa" containerName="revoke-keys"
Oct 31 02:04:41.971067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:41.969277    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad39557d-531b-486d-90f8-c63f8b8e12fa" containerName="revoke-keys"
Oct 31 02:04:41.971067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:41.969586    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad39557d-531b-486d-90f8-c63f8b8e12fa" containerName="revoke-keys"
Oct 31 02:04:41.976330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:41.976307    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.001901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.001873    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:04:42.002262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.002242    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:04:42.226118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.222467    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68f815a-98a1-4214-baba-30b97e893687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.298369 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.298332    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68f815a-98a1-4214-baba-30b97e893687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.298576 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.298541    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shk6k\" (UniqueName: \"kubernetes.io/projected/f68f815a-98a1-4214-baba-30b97e893687-kube-api-access-shk6k\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.300085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.299996    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"]
Oct 31 02:04:42.492182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.488245    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68f815a-98a1-4214-baba-30b97e893687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.499737 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.499709    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68f815a-98a1-4214-baba-30b97e893687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.500136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.500118    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shk6k\" (UniqueName: \"kubernetes.io/projected/f68f815a-98a1-4214-baba-30b97e893687-kube-api-access-shk6k\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.501082 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:04:42.497747    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:04:42.501781 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.501755    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:04:42.502803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.502779    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68f815a-98a1-4214-baba-30b97e893687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.515341 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.514129    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68f815a-98a1-4214-baba-30b97e893687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.525249 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.525204    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shk6k\" (UniqueName: \"kubernetes.io/projected/f68f815a-98a1-4214-baba-30b97e893687-kube-api-access-shk6k\") pod \"maas-api-revoke-sub-d66e0592-jmhmm\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.605685 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.605653    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:04:42.613921 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.613886    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:42.914147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.914112    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"]
Oct 31 02:04:42.918731 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:04:42.918699    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68f815a_98a1_4214_baba_30b97e893687.slice/crio-13c1278c109bb316cd54c8a6e55abc20db49b5cd9b0d3591e2256f4938edca9e WatchSource:0}: Error finding container 13c1278c109bb316cd54c8a6e55abc20db49b5cd9b0d3591e2256f4938edca9e: Status 404 returned error can't find the container with id 13c1278c109bb316cd54c8a6e55abc20db49b5cd9b0d3591e2256f4938edca9e
Oct 31 02:04:42.960016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:42.959982    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm" event={"ID":"f68f815a-98a1-4214-baba-30b97e893687","Type":"ContainerStarted","Data":"13c1278c109bb316cd54c8a6e55abc20db49b5cd9b0d3591e2256f4938edca9e"}
Oct 31 02:04:43.974772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:43.974738    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm" event={"ID":"f68f815a-98a1-4214-baba-30b97e893687","Type":"ContainerStarted","Data":"86262cae1db54a5d788d5daa4bcd3087dc2edc96c14a985e35bbb65b9861079c"}
Oct 31 02:04:43.994387 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:43.994335    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm" podStartSLOduration=2.994323733 podStartE2EDuration="2.994323733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:43.992206891 +0000 UTC m=+2641.940855285" watchObservedRunningTime="2026-10-31 02:04:43.994323733 +0000 UTC m=+2641.942972125"
Oct 31 02:04:44.986721 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:44.986686    5293 generic.go:356] "Generic (PLEG): container finished" podID="f68f815a-98a1-4214-baba-30b97e893687" containerID="86262cae1db54a5d788d5daa4bcd3087dc2edc96c14a985e35bbb65b9861079c" exitCode=0
Oct 31 02:04:44.987252 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:44.986814    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm" event={"ID":"f68f815a-98a1-4214-baba-30b97e893687","Type":"ContainerDied","Data":"86262cae1db54a5d788d5daa4bcd3087dc2edc96c14a985e35bbb65b9861079c"}
Oct 31 02:04:46.193819 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.193793    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:46.238308 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.238272    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68f815a-98a1-4214-baba-30b97e893687-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68f815a-98a1-4214-baba-30b97e893687-maas-api-xks-ca\") pod \"f68f815a-98a1-4214-baba-30b97e893687\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") "
Oct 31 02:04:46.238624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.238368    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68f815a-98a1-4214-baba-30b97e893687-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68f815a-98a1-4214-baba-30b97e893687-maas-api-openshift-ca\") pod \"f68f815a-98a1-4214-baba-30b97e893687\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") "
Oct 31 02:04:46.238717 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.238681    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68f815a-98a1-4214-baba-30b97e893687-kube-api-access-shk6k\" (UniqueName: \"kubernetes.io/projected/f68f815a-98a1-4214-baba-30b97e893687-kube-api-access-shk6k\") pod \"f68f815a-98a1-4214-baba-30b97e893687\" (UID: \"f68f815a-98a1-4214-baba-30b97e893687\") "
Oct 31 02:04:46.239090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.239052    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68f815a-98a1-4214-baba-30b97e893687-maas-api-openshift-ca" pod "f68f815a-98a1-4214-baba-30b97e893687" (UID: "f68f815a-98a1-4214-baba-30b97e893687"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:46.239796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.239771    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68f815a-98a1-4214-baba-30b97e893687-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:46.249146 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.246552    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68f815a-98a1-4214-baba-30b97e893687-kube-api-access-shk6k" pod "f68f815a-98a1-4214-baba-30b97e893687" (UID: "f68f815a-98a1-4214-baba-30b97e893687"). InnerVolumeSpecName "kube-api-access-shk6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:46.249146 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.248812    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68f815a-98a1-4214-baba-30b97e893687-maas-api-xks-ca" pod "f68f815a-98a1-4214-baba-30b97e893687" (UID: "f68f815a-98a1-4214-baba-30b97e893687"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:46.342048 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.342012    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shk6k\" (UniqueName: \"kubernetes.io/projected/f68f815a-98a1-4214-baba-30b97e893687-kube-api-access-shk6k\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:46.342266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:46.342063    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68f815a-98a1-4214-baba-30b97e893687-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:47.153065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.143097    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm" event={"ID":"f68f815a-98a1-4214-baba-30b97e893687","Type":"ContainerDied","Data":"13c1278c109bb316cd54c8a6e55abc20db49b5cd9b0d3591e2256f4938edca9e"}
Oct 31 02:04:47.153431 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.153399    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13c1278c109bb316cd54c8a6e55abc20db49b5cd9b0d3591e2256f4938edca9e"
Oct 31 02:04:47.153644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.153629    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"
Oct 31 02:04:47.444177 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.444101    5293 scope.go:122] "RemoveContainer" containerID="1864c4be79b0c17ca6e7e4332f44275af54e14523509a40561f1d366a09621da"
Oct 31 02:04:47.635125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.631343    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"]
Oct 31 02:04:47.635125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.632718    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f68f815a-98a1-4214-baba-30b97e893687" containerName="revoke-keys"
Oct 31 02:04:47.635125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.632743    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68f815a-98a1-4214-baba-30b97e893687" containerName="revoke-keys"
Oct 31 02:04:47.635125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.633024    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68f815a-98a1-4214-baba-30b97e893687" containerName="revoke-keys"
Oct 31 02:04:47.667065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.639933    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:47.667065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.665234    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:04:47.667065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.665373    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:04:47.681001 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.680967    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"]
Oct 31 02:04:47.697111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.693559    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:47.705533 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.705503    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbgsb\" (UniqueName: \"kubernetes.io/projected/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-kube-api-access-fbgsb\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:47.706411 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.706386    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:47.927522 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.927481    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbgsb\" (UniqueName: \"kubernetes.io/projected/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-kube-api-access-fbgsb\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:47.928368 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.928341    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:47.933288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.933258    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:47.940511 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.940490    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:47.990929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:47.989522    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbgsb\" (UniqueName: \"kubernetes.io/projected/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-kube-api-access-fbgsb\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:48.040555 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:48.040517    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1989431f-nxmrf\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:48.293477 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:48.293402    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:48.658862 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:04:48.658831    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cf7cf9_7810_4084_b25e_b2eb7768bf0b.slice/crio-1f7a12728012e17b2a9ce77c848a09635b4509583c108b777cdca8f835c0f6f2 WatchSource:0}: Error finding container 1f7a12728012e17b2a9ce77c848a09635b4509583c108b777cdca8f835c0f6f2: Status 404 returned error can't find the container with id 1f7a12728012e17b2a9ce77c848a09635b4509583c108b777cdca8f835c0f6f2
Oct 31 02:04:48.659304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:48.658967    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"]
Oct 31 02:04:49.167913 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:49.167792    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf" event={"ID":"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b","Type":"ContainerStarted","Data":"1f7a12728012e17b2a9ce77c848a09635b4509583c108b777cdca8f835c0f6f2"}
Oct 31 02:04:50.181255 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:50.181223    5293 generic.go:356] "Generic (PLEG): container finished" podID="d3cf7cf9-7810-4084-b25e-b2eb7768bf0b" containerID="fa5adc20eea9ccbd155114a9c0f6c318dd7d0b64d1e5923430d77c124c2f3984" exitCode=0
Oct 31 02:04:50.181725 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:50.181300    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf" event={"ID":"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b","Type":"ContainerDied","Data":"fa5adc20eea9ccbd155114a9c0f6c318dd7d0b64d1e5923430d77c124c2f3984"}
Oct 31 02:04:51.396584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.396560    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:51.428641 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.428603    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-openshift-ca\") pod \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") "
Oct 31 02:04:51.428818 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.428657    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-xks-ca\") pod \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") "
Oct 31 02:04:51.428818 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.428770    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-kube-api-access-fbgsb\" (UniqueName: \"kubernetes.io/projected/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-kube-api-access-fbgsb\") pod \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\" (UID: \"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b\") "
Oct 31 02:04:51.429368 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.429338    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-openshift-ca" pod "d3cf7cf9-7810-4084-b25e-b2eb7768bf0b" (UID: "d3cf7cf9-7810-4084-b25e-b2eb7768bf0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:51.430134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.429966    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:51.431448 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.431422    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-xks-ca" pod "d3cf7cf9-7810-4084-b25e-b2eb7768bf0b" (UID: "d3cf7cf9-7810-4084-b25e-b2eb7768bf0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:51.431630 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.431610    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-kube-api-access-fbgsb" pod "d3cf7cf9-7810-4084-b25e-b2eb7768bf0b" (UID: "d3cf7cf9-7810-4084-b25e-b2eb7768bf0b"). InnerVolumeSpecName "kube-api-access-fbgsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:51.532291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.532135    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:51.532291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:51.532166    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbgsb\" (UniqueName: \"kubernetes.io/projected/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b-kube-api-access-fbgsb\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:52.035668 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:52.035632    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"]
Oct 31 02:04:52.040584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:52.040554    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad4bbf3-zl6zp"]
Oct 31 02:04:52.178587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:52.178556    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e2a839-b9a2-43c3-ad5b-b7aef672b314" path="/var/lib/kubelet/pods/61e2a839-b9a2-43c3-ad5b-b7aef672b314/volumes"
Oct 31 02:04:52.199976 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:52.199948    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"
Oct 31 02:04:52.200161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:52.199976    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf" event={"ID":"d3cf7cf9-7810-4084-b25e-b2eb7768bf0b","Type":"ContainerDied","Data":"1f7a12728012e17b2a9ce77c848a09635b4509583c108b777cdca8f835c0f6f2"}
Oct 31 02:04:52.200161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:52.200028    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f7a12728012e17b2a9ce77c848a09635b4509583c108b777cdca8f835c0f6f2"
Oct 31 02:04:53.989462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:53.987136    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:04:54.074170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.074117    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"]
Oct 31 02:04:54.075962 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.075929    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d3cf7cf9-7810-4084-b25e-b2eb7768bf0b" containerName="revoke-keys"
Oct 31 02:04:54.088157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.088110    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cf7cf9-7810-4084-b25e-b2eb7768bf0b" containerName="revoke-keys"
Oct 31 02:04:54.104090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.088821    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cf7cf9-7810-4084-b25e-b2eb7768bf0b" containerName="revoke-keys"
Oct 31 02:04:54.284466 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.200512    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:04:54.284466 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.282326    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:04:54.316117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.286150    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.316117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.286328    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtphp\" (UniqueName: \"kubernetes.io/projected/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-kube-api-access-dtphp\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.316117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.286413    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.317280 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.317254    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.366123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.341690    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:04:54.366123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.341907    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:04:54.396955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.395503    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.396955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.395872    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.396955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.395981    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtphp\" (UniqueName: \"kubernetes.io/projected/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-kube-api-access-dtphp\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.399063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.398023    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.478064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.477129    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.512164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.512132    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtphp\" (UniqueName: \"kubernetes.io/projected/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-kube-api-access-dtphp\") pod \"maas-api-revoke-sub-fac276b8-ts9rc\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.543186 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.543108    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:54.577118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.565123    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:04:54.627834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.627747    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"]
Oct 31 02:04:54.684153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.676478    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded"
Oct 31 02:04:54.861468 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:54.861385    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"]
Oct 31 02:04:55.364570 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:55.362910    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"]
Oct 31 02:04:55.462404 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:55.455533    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc" event={"ID":"8c96b803-3046-4e4a-a5a5-76c1a2adde1a","Type":"ContainerStarted","Data":"84ded1a557cafe692dcab2f3c0b5079400c05dfc077c0a383622553e156e8e38"}
Oct 31 02:04:56.499635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:56.499545    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc" event={"ID":"8c96b803-3046-4e4a-a5a5-76c1a2adde1a","Type":"ContainerStarted","Data":"c5b3c23452abee11c71a2c9dd39f1cc4c73cb091a7f835d57cddd81949f0a855"}
Oct 31 02:04:56.521061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:56.520997    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc" podStartSLOduration=3.52098404 podStartE2EDuration="3.52098404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:56.517934605 +0000 UTC m=+2654.466583003" watchObservedRunningTime="2026-10-31 02:04:56.52098404 +0000 UTC m=+2654.469632459"
Oct 31 02:04:57.511981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:57.511940    5293 generic.go:356] "Generic (PLEG): container finished" podID="8c96b803-3046-4e4a-a5a5-76c1a2adde1a" containerID="c5b3c23452abee11c71a2c9dd39f1cc4c73cb091a7f835d57cddd81949f0a855" exitCode=0
Oct 31 02:04:57.512445 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:57.512024    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc" event={"ID":"8c96b803-3046-4e4a-a5a5-76c1a2adde1a","Type":"ContainerDied","Data":"c5b3c23452abee11c71a2c9dd39f1cc4c73cb091a7f835d57cddd81949f0a855"}
Oct 31 02:04:58.717891 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.717866    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:58.773057 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.772801    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-xks-ca\") pod \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") "
Oct 31 02:04:58.773057 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.772985    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-openshift-ca\") pod \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") "
Oct 31 02:04:58.774113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.773395    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-kube-api-access-dtphp\" (UniqueName: \"kubernetes.io/projected/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-kube-api-access-dtphp\") pod \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\" (UID: \"8c96b803-3046-4e4a-a5a5-76c1a2adde1a\") "
Oct 31 02:04:58.775796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.775692    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-openshift-ca" pod "8c96b803-3046-4e4a-a5a5-76c1a2adde1a" (UID: "8c96b803-3046-4e4a-a5a5-76c1a2adde1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:58.779028 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.778990    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-xks-ca" pod "8c96b803-3046-4e4a-a5a5-76c1a2adde1a" (UID: "8c96b803-3046-4e4a-a5a5-76c1a2adde1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:58.794255 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.794206    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"]
Oct 31 02:04:58.810744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.810711    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-kube-api-access-dtphp" pod "8c96b803-3046-4e4a-a5a5-76c1a2adde1a" (UID: "8c96b803-3046-4e4a-a5a5-76c1a2adde1a"). InnerVolumeSpecName "kube-api-access-dtphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:58.875829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.875799    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtphp\" (UniqueName: \"kubernetes.io/projected/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-kube-api-access-dtphp\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:58.875829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.875829    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:58.876033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:58.875839    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c96b803-3046-4e4a-a5a5-76c1a2adde1a-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:59.536905 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:59.536874    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"
Oct 31 02:04:59.537144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:59.536875    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc" event={"ID":"8c96b803-3046-4e4a-a5a5-76c1a2adde1a","Type":"ContainerDied","Data":"84ded1a557cafe692dcab2f3c0b5079400c05dfc077c0a383622553e156e8e38"}
Oct 31 02:04:59.537144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:59.537012    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84ded1a557cafe692dcab2f3c0b5079400c05dfc077c0a383622553e156e8e38"
Oct 31 02:04:59.862259 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:04:59.862175    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz" podUID="d1282bdd-59fa-4e77-87cc-84182fd71d46" containerName="payload-processing" containerID="cri-o://de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73" gracePeriod=25
Oct 31 02:05:00.181408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.181381    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:05:00.216457 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.216421    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"]
Oct 31 02:05:00.218097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.217834    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d1282bdd-59fa-4e77-87cc-84182fd71d46" containerName="payload-processing"
Oct 31 02:05:00.218097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.217874    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1282bdd-59fa-4e77-87cc-84182fd71d46" containerName="payload-processing"
Oct 31 02:05:00.218097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.217942    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8c96b803-3046-4e4a-a5a5-76c1a2adde1a" containerName="revoke-keys"
Oct 31 02:05:00.218097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.217952    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c96b803-3046-4e4a-a5a5-76c1a2adde1a" containerName="revoke-keys"
Oct 31 02:05:00.218455 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.218219    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c96b803-3046-4e4a-a5a5-76c1a2adde1a" containerName="revoke-keys"
Oct 31 02:05:00.218455 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.218254    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1282bdd-59fa-4e77-87cc-84182fd71d46" containerName="payload-processing"
Oct 31 02:05:00.225718 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.225697    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.231419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.231361    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"agc-cred-1ec50ba6b83f\""
Oct 31 02:05:00.235929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.235747    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"]
Oct 31 02:05:00.242268 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.242236    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-kube-api-access-56b6w\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-kube-api-access-56b6w\") pod \"d1282bdd-59fa-4e77-87cc-84182fd71d46\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") "
Oct 31 02:05:00.242476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.242337    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-provider-credentials\") pod \"d1282bdd-59fa-4e77-87cc-84182fd71d46\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") "
Oct 31 02:05:00.242476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.242438    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-plugins-config-volume\") pod \"d1282bdd-59fa-4e77-87cc-84182fd71d46\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") "
Oct 31 02:05:00.242854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.242806    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-routing-overlay-volume\") pod \"d1282bdd-59fa-4e77-87cc-84182fd71d46\" (UID: \"d1282bdd-59fa-4e77-87cc-84182fd71d46\") "
Oct 31 02:05:00.244210 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.243540    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-routing-overlay-volume" pod "d1282bdd-59fa-4e77-87cc-84182fd71d46" (UID: "d1282bdd-59fa-4e77-87cc-84182fd71d46"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:00.247190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.247150    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-plugins-config-volume" pod "d1282bdd-59fa-4e77-87cc-84182fd71d46" (UID: "d1282bdd-59fa-4e77-87cc-84182fd71d46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:00.249990 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.249956    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.250343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.250319    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.250452 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.250426    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8zh\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-kube-api-access-kz8zh\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.250528 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.250507    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.250771 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.250754    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:00.250891 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.250778    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1282bdd-59fa-4e77-87cc-84182fd71d46-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:00.256825 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.256799    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-provider-credentials" pod "d1282bdd-59fa-4e77-87cc-84182fd71d46" (UID: "d1282bdd-59fa-4e77-87cc-84182fd71d46"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:00.258483 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.258460    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-kube-api-access-56b6w" pod "d1282bdd-59fa-4e77-87cc-84182fd71d46" (UID: "d1282bdd-59fa-4e77-87cc-84182fd71d46"). InnerVolumeSpecName "kube-api-access-56b6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:00.366063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.356350    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.366063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.356665    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.366063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.356756    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8zh\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-kube-api-access-kz8zh\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.366063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.356833    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.366063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.357395    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56b6w\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-kube-api-access-56b6w\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:00.366063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.357425    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1282bdd-59fa-4e77-87cc-84182fd71d46-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:00.423063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.412450    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.435844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.435233    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8zh\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-kube-api-access-kz8zh\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.436805 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.436774    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.456753 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.456028    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.549950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.549912    5293 generic.go:356] "Generic (PLEG): container finished" podID="d1282bdd-59fa-4e77-87cc-84182fd71d46" containerID="de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73" exitCode=0
Oct 31 02:05:00.549950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.549954    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"
Oct 31 02:05:00.550298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.550080    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz" event={"ID":"d1282bdd-59fa-4e77-87cc-84182fd71d46","Type":"ContainerDied","Data":"de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73"}
Oct 31 02:05:00.550298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.550133    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz" event={"ID":"d1282bdd-59fa-4e77-87cc-84182fd71d46","Type":"ContainerDied","Data":"4c4851aa09e3f00b40d68fc76d303b98796793e012f5d6c154ac005c6eaeba95"}
Oct 31 02:05:00.550298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.550155    5293 scope.go:122] "RemoveContainer" containerID="de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73"
Oct 31 02:05:00.563384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.562708    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:00.568262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.568241    5293 scope.go:122] "RemoveContainer" containerID="de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73"
Oct 31 02:05:00.568572 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:00.568551    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73\": container with ID starting with de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73 not found: ID does not exist" containerID="de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73"
Oct 31 02:05:00.568649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.568580    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73"} err="failed to get container status \"de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73\": rpc error: code = NotFound desc = could not find container \"de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73\": container with ID starting with de6617ebf6be4537001568d682769200aec4f66a5bf5eebc485010dcc02eee73 not found: ID does not exist"
Oct 31 02:05:00.581088 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.581067    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"]
Oct 31 02:05:00.585984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.585962    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-7bcfnglsz"]
Oct 31 02:05:00.853869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:00.853834    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"]
Oct 31 02:05:00.854413 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:05:00.854388    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c5673b_5d4c_4518_9088_d0f9304638a5.slice/crio-3a37606926a5f755347d7169dc1849941f76036de64553a8a707e61f823ff88d WatchSource:0}: Error finding container 3a37606926a5f755347d7169dc1849941f76036de64553a8a707e61f823ff88d: Status 404 returned error can't find the container with id 3a37606926a5f755347d7169dc1849941f76036de64553a8a707e61f823ff88d
Oct 31 02:05:01.564538 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:01.564501    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn" event={"ID":"b1c5673b-5d4c-4518-9088-d0f9304638a5","Type":"ContainerStarted","Data":"66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c"}
Oct 31 02:05:01.564979 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:01.564603    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:01.564979 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:01.564628    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn" event={"ID":"b1c5673b-5d4c-4518-9088-d0f9304638a5","Type":"ContainerStarted","Data":"3a37606926a5f755347d7169dc1849941f76036de64553a8a707e61f823ff88d"}
Oct 31 02:05:01.588486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:01.588440    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn" podStartSLOduration=25.588426885 podStartE2EDuration="25.588426885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:01.582950531 +0000 UTC m=+2659.531598923" watchObservedRunningTime="2026-10-31 02:05:01.588426885 +0000 UTC m=+2659.537075276"
Oct 31 02:05:02.177057 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:02.177016    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1282bdd-59fa-4e77-87cc-84182fd71d46" path="/var/lib/kubelet/pods/d1282bdd-59fa-4e77-87cc-84182fd71d46/volumes"
Oct 31 02:05:03.227481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.227029    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"]
Oct 31 02:05:03.298102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.297626    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.306065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.305573    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"]
Oct 31 02:05:03.315128 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.314807    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:05:03.315128 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.314890    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:05:03.342943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.342908    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6855f\" (UniqueName: \"kubernetes.io/projected/609e08ae-ddf0-43a0-8854-110ccdcb2f02-kube-api-access-6855f\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.343134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.343096    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.343134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.343127    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.445522 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.445480    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6855f\" (UniqueName: \"kubernetes.io/projected/609e08ae-ddf0-43a0-8854-110ccdcb2f02-kube-api-access-6855f\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.445877 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.445695    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.446288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.446229    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.446803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.446781    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.448781 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.448755    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.457762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.457741    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6855f\" (UniqueName: \"kubernetes.io/projected/609e08ae-ddf0-43a0-8854-110ccdcb2f02-kube-api-access-6855f\") pod \"maas-api-revoke-sub-92535836-bfgs6\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.638944 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.638866    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:03.798350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:03.798287    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs" podUID="091dea8e-578a-4fb6-9fa1-16302e983ac5" containerName="payload-processing" containerID="cri-o://05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe" gracePeriod=25
Oct 31 02:05:04.007932 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:05:04.007225    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609e08ae_ddf0_43a0_8854_110ccdcb2f02.slice/crio-6bc024dd2104cff54dd39a3fb298298719537801c951bb7464bd63ad25654f17 WatchSource:0}: Error finding container 6bc024dd2104cff54dd39a3fb298298719537801c951bb7464bd63ad25654f17: Status 404 returned error can't find the container with id 6bc024dd2104cff54dd39a3fb298298719537801c951bb7464bd63ad25654f17
Oct 31 02:05:04.029303 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.028923    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"]
Oct 31 02:05:04.036812 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.036778    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"]
Oct 31 02:05:04.044124 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.042682    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deaa4bc-f5bbf"]
Oct 31 02:05:04.163459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.163430    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:05:04.178505 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.178480    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852" path="/var/lib/kubelet/pods/1da82c0d-cf6f-4b2e-8dbc-9e97fbfec852/volumes"
Oct 31 02:05:04.223841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.222273    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"]
Oct 31 02:05:04.223841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.223752    5293 cpu_manager.go:414] "containerMap: removing container" podUID="091dea8e-578a-4fb6-9fa1-16302e983ac5" containerName="payload-processing"
Oct 31 02:05:04.223841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.223773    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091dea8e-578a-4fb6-9fa1-16302e983ac5" containerName="payload-processing"
Oct 31 02:05:04.224129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.223975    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="091dea8e-578a-4fb6-9fa1-16302e983ac5" containerName="payload-processing"
Oct 31 02:05:04.254364 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.254336    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.258139 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.258104    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"]
Oct 31 02:05:04.258693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.258673    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-09706649fc65\""
Oct 31 02:05:04.258824 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.258782    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-e2e--c450cd8d-dockercfg-2j4lf\""
Oct 31 02:05:04.260972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.260942    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0b6abf\""
Oct 31 02:05:04.261112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.261090    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"openshift-service-ca.crt\""
Oct 31 02:05:04.261196 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.260968    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"routing-overlay\""
Oct 31 02:05:04.261269 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.261235    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"kube-root-ca.crt\""
Oct 31 02:05:04.347070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.347026    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-plugins-config-volume\") pod \"091dea8e-578a-4fb6-9fa1-16302e983ac5\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") "
Oct 31 02:05:04.347262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.347167    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-kube-api-access-9bmxw\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-kube-api-access-9bmxw\") pod \"091dea8e-578a-4fb6-9fa1-16302e983ac5\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") "
Oct 31 02:05:04.347262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.347255    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-routing-overlay-volume\") pod \"091dea8e-578a-4fb6-9fa1-16302e983ac5\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") "
Oct 31 02:05:04.347597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.347567    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-plugins-config-volume" pod "091dea8e-578a-4fb6-9fa1-16302e983ac5" (UID: "091dea8e-578a-4fb6-9fa1-16302e983ac5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:04.347683 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.347572    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-routing-overlay-volume" pod "091dea8e-578a-4fb6-9fa1-16302e983ac5" (UID: "091dea8e-578a-4fb6-9fa1-16302e983ac5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:04.347683 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.347637    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-provider-credentials\") pod \"091dea8e-578a-4fb6-9fa1-16302e983ac5\" (UID: \"091dea8e-578a-4fb6-9fa1-16302e983ac5\") "
Oct 31 02:05:04.348843 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.348767    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:04.348843 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.348793    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091dea8e-578a-4fb6-9fa1-16302e983ac5-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:04.350672 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.350648    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-kube-api-access-9bmxw" pod "091dea8e-578a-4fb6-9fa1-16302e983ac5" (UID: "091dea8e-578a-4fb6-9fa1-16302e983ac5"). InnerVolumeSpecName "kube-api-access-9bmxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:04.350777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.350757    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-provider-credentials" pod "091dea8e-578a-4fb6-9fa1-16302e983ac5" (UID: "091dea8e-578a-4fb6-9fa1-16302e983ac5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:04.449976 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.449947    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.450113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.450076    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.450194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.450117    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jf5m\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-kube-api-access-8jf5m\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.450383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.450361    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.450698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.450675    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bmxw\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-kube-api-access-9bmxw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:04.450698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.450699    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/091dea8e-578a-4fb6-9fa1-16302e983ac5-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:04.552207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.552125    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.552207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.552183    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf5m\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-kube-api-access-8jf5m\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.552416 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.552399    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.552703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.552688    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.552919 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.552882    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.553433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.553407    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.555227 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.555203    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.565359 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.565339    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf5m\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-kube-api-access-8jf5m\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.579400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.579375    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:04.746092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.741740    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6" event={"ID":"609e08ae-ddf0-43a0-8854-110ccdcb2f02","Type":"ContainerStarted","Data":"1aaa211dd5f043c5b151b889bdb9948432b09994f5f178f6b3c3d51e98249f60"}
Oct 31 02:05:04.746092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.741815    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6" event={"ID":"609e08ae-ddf0-43a0-8854-110ccdcb2f02","Type":"ContainerStarted","Data":"6bc024dd2104cff54dd39a3fb298298719537801c951bb7464bd63ad25654f17"}
Oct 31 02:05:04.788527 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.788404    5293 generic.go:356] "Generic (PLEG): container finished" podID="091dea8e-578a-4fb6-9fa1-16302e983ac5" containerID="05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe" exitCode=0
Oct 31 02:05:04.788527 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.788510    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs" event={"ID":"091dea8e-578a-4fb6-9fa1-16302e983ac5","Type":"ContainerDied","Data":"05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe"}
Oct 31 02:05:04.788756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.788569    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs" event={"ID":"091dea8e-578a-4fb6-9fa1-16302e983ac5","Type":"ContainerDied","Data":"0957ca371e06b45a89c589259e7cdfd3454375991151599b12714de09cdcb949"}
Oct 31 02:05:04.788756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.788596    5293 scope.go:122] "RemoveContainer" containerID="05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe"
Oct 31 02:05:04.788871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.788817    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"
Oct 31 02:05:04.798400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.796062    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6" podStartSLOduration=1.79602061 podStartE2EDuration="1.79602061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:04.79306219 +0000 UTC m=+2662.741710582" watchObservedRunningTime="2026-10-31 02:05:04.79602061 +0000 UTC m=+2662.744669002"
Oct 31 02:05:04.834432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.834410    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"]
Oct 31 02:05:04.834701 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.834684    5293 scope.go:122] "RemoveContainer" containerID="05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe"
Oct 31 02:05:04.835638 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:04.835609    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe\": container with ID starting with 05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe not found: ID does not exist" containerID="05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe"
Oct 31 02:05:04.835747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.835649    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe"} err="failed to get container status \"05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe\": rpc error: code = NotFound desc = could not find container \"05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe\": container with ID starting with 05ff290272a0890700086c7a125ad466f55a6c85a136a38c4950b647c91746fe not found: ID does not exist"
Oct 31 02:05:04.841587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:04.841566    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-797fz79gs"]
Oct 31 02:05:05.038880 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:05.037519    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"]
Oct 31 02:05:05.064919 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:05.064886    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"]
Oct 31 02:05:05.943624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:05.943129    5293 generic.go:356] "Generic (PLEG): container finished" podID="609e08ae-ddf0-43a0-8854-110ccdcb2f02" containerID="1aaa211dd5f043c5b151b889bdb9948432b09994f5f178f6b3c3d51e98249f60" exitCode=0
Oct 31 02:05:05.943624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:05.943231    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6" event={"ID":"609e08ae-ddf0-43a0-8854-110ccdcb2f02","Type":"ContainerDied","Data":"1aaa211dd5f043c5b151b889bdb9948432b09994f5f178f6b3c3d51e98249f60"}
Oct 31 02:05:05.951988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:05.951371    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9" event={"ID":"997cb194-76f1-4567-b54c-bd8f8f690bcb","Type":"ContainerStarted","Data":"45ea8398d6b204a7b8ba650a9acecb830a65038dbad491922e0b6943c0ccf28f"}
Oct 31 02:05:05.951988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:05.951435    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9" event={"ID":"997cb194-76f1-4567-b54c-bd8f8f690bcb","Type":"ContainerStarted","Data":"cbe8b9652e7f46e6d3abe95f50ecb16bb14e4362f73833deea66de4c0148f95f"}
Oct 31 02:05:05.951988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:05.951766    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:05.993110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:05.993063    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9" podStartSLOduration=43.993029382 podStartE2EDuration="43.993029382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:05.98834289 +0000 UTC m=+2663.936991281" watchObservedRunningTime="2026-10-31 02:05:05.993029382 +0000 UTC m=+2663.941677774"
Oct 31 02:05:06.191430 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:06.191397    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091dea8e-578a-4fb6-9fa1-16302e983ac5" path="/var/lib/kubelet/pods/091dea8e-578a-4fb6-9fa1-16302e983ac5/volumes"
Oct 31 02:05:07.180946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.180920    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:07.233331 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.233235    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-openshift-ca\") pod \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") "
Oct 31 02:05:07.233480 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.233433    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-xks-ca\") pod \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") "
Oct 31 02:05:07.233703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.233685    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609e08ae-ddf0-43a0-8854-110ccdcb2f02-kube-api-access-6855f\" (UniqueName: \"kubernetes.io/projected/609e08ae-ddf0-43a0-8854-110ccdcb2f02-kube-api-access-6855f\") pod \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\" (UID: \"609e08ae-ddf0-43a0-8854-110ccdcb2f02\") "
Oct 31 02:05:07.235940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.235899    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-openshift-ca" pod "609e08ae-ddf0-43a0-8854-110ccdcb2f02" (UID: "609e08ae-ddf0-43a0-8854-110ccdcb2f02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:07.239837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.238154    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609e08ae-ddf0-43a0-8854-110ccdcb2f02-kube-api-access-6855f" pod "609e08ae-ddf0-43a0-8854-110ccdcb2f02" (UID: "609e08ae-ddf0-43a0-8854-110ccdcb2f02"). InnerVolumeSpecName "kube-api-access-6855f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:07.248202 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.240217    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-xks-ca" pod "609e08ae-ddf0-43a0-8854-110ccdcb2f02" (UID: "609e08ae-ddf0-43a0-8854-110ccdcb2f02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:05:07.337081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.337022    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:07.337081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.337071    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6855f\" (UniqueName: \"kubernetes.io/projected/609e08ae-ddf0-43a0-8854-110ccdcb2f02-kube-api-access-6855f\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:07.337081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.337089    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609e08ae-ddf0-43a0-8854-110ccdcb2f02-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:07.974086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.974055    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"
Oct 31 02:05:07.974086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.974066    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6" event={"ID":"609e08ae-ddf0-43a0-8854-110ccdcb2f02","Type":"ContainerDied","Data":"6bc024dd2104cff54dd39a3fb298298719537801c951bb7464bd63ad25654f17"}
Oct 31 02:05:07.974304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:07.974144    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc024dd2104cff54dd39a3fb298298719537801c951bb7464bd63ad25654f17"
Oct 31 02:05:10.958079 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:10.956852    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9" podUID="997cb194-76f1-4567-b54c-bd8f8f690bcb" containerName="payload-processing" containerID="cri-o://45ea8398d6b204a7b8ba650a9acecb830a65038dbad491922e0b6943c0ccf28f" gracePeriod=25
Oct 31 02:05:11.177847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:11.177696    5293 generic.go:356] "Generic (PLEG): container finished" podID="997cb194-76f1-4567-b54c-bd8f8f690bcb" containerID="45ea8398d6b204a7b8ba650a9acecb830a65038dbad491922e0b6943c0ccf28f" exitCode=0
Oct 31 02:05:11.177847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:11.177784    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9" event={"ID":"997cb194-76f1-4567-b54c-bd8f8f690bcb","Type":"ContainerDied","Data":"45ea8398d6b204a7b8ba650a9acecb830a65038dbad491922e0b6943c0ccf28f"}
Oct 31 02:05:12.081460 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:12.081418    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:05:12.174141 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:12.173843    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997cb194_76f1_4567_b54c_bd8f8f690bcb.slice/crio-conmon-45ea8398d6b204a7b8ba650a9acecb830a65038dbad491922e0b6943c0ccf28f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997cb194_76f1_4567_b54c_bd8f8f690bcb.slice/crio-45ea8398d6b204a7b8ba650a9acecb830a65038dbad491922e0b6943c0ccf28f.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 02:05:12.281401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.281372    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:12.424598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.424557    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-provider-credentials\") pod \"997cb194-76f1-4567-b54c-bd8f8f690bcb\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") "
Oct 31 02:05:12.424870 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.424848    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-plugins-config-volume\") pod \"997cb194-76f1-4567-b54c-bd8f8f690bcb\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") "
Oct 31 02:05:12.424967 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.424950    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-kube-api-access-8jf5m\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-kube-api-access-8jf5m\") pod \"997cb194-76f1-4567-b54c-bd8f8f690bcb\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") "
Oct 31 02:05:12.425112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.425095    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-routing-overlay-volume\") pod \"997cb194-76f1-4567-b54c-bd8f8f690bcb\" (UID: \"997cb194-76f1-4567-b54c-bd8f8f690bcb\") "
Oct 31 02:05:12.425502 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.425427    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-plugins-config-volume" pod "997cb194-76f1-4567-b54c-bd8f8f690bcb" (UID: "997cb194-76f1-4567-b54c-bd8f8f690bcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:12.425897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.425872    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-routing-overlay-volume" pod "997cb194-76f1-4567-b54c-bd8f8f690bcb" (UID: "997cb194-76f1-4567-b54c-bd8f8f690bcb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:12.425993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.425944    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:12.427126 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.427102    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-provider-credentials" pod "997cb194-76f1-4567-b54c-bd8f8f690bcb" (UID: "997cb194-76f1-4567-b54c-bd8f8f690bcb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:12.427604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.427570    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-kube-api-access-8jf5m" pod "997cb194-76f1-4567-b54c-bd8f8f690bcb" (UID: "997cb194-76f1-4567-b54c-bd8f8f690bcb"). InnerVolumeSpecName "kube-api-access-8jf5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:12.527872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.527829    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jf5m\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-kube-api-access-8jf5m\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:12.527872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.527871    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/997cb194-76f1-4567-b54c-bd8f8f690bcb-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:12.528167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.527888    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/997cb194-76f1-4567-b54c-bd8f8f690bcb-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:12.647457 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.645791    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:12.724562 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:12.724477    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"]
Oct 31 02:05:13.324162 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:13.324121    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9" event={"ID":"997cb194-76f1-4567-b54c-bd8f8f690bcb","Type":"ContainerDied","Data":"cbe8b9652e7f46e6d3abe95f50ecb16bb14e4362f73833deea66de4c0148f95f"}
Oct 31 02:05:13.324846 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:13.324818    5293 scope.go:122] "RemoveContainer" containerID="45ea8398d6b204a7b8ba650a9acecb830a65038dbad491922e0b6943c0ccf28f"
Oct 31 02:05:13.325224 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:13.325202    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"
Oct 31 02:05:13.471353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:13.471321    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"]
Oct 31 02:05:13.476294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:13.476227    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-7bd7t76n9"]
Oct 31 02:05:14.177326 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:14.177296    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997cb194-76f1-4567-b54c-bd8f8f690bcb" path="/var/lib/kubelet/pods/997cb194-76f1-4567-b54c-bd8f8f690bcb/volumes"
Oct 31 02:05:15.782594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:15.760802    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:05:15.795189 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:15.795140    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:15.866170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:15.866118    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:16.153635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:16.153594    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:16.154866 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:16.153940    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:05:16.264243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:16.264184    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:16.791162 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:16.791119    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:16.893646 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:16.893602    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:05:17.168138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.168107    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"]
Oct 31 02:05:17.187416 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.169807    5293 cpu_manager.go:414] "containerMap: removing container" podUID="997cb194-76f1-4567-b54c-bd8f8f690bcb" containerName="payload-processing"
Oct 31 02:05:17.189729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.189692    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997cb194-76f1-4567-b54c-bd8f8f690bcb" containerName="payload-processing"
Oct 31 02:05:17.190012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.189997    5293 cpu_manager.go:414] "containerMap: removing container" podUID="609e08ae-ddf0-43a0-8854-110ccdcb2f02" containerName="revoke-keys"
Oct 31 02:05:17.190142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.190129    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609e08ae-ddf0-43a0-8854-110ccdcb2f02" containerName="revoke-keys"
Oct 31 02:05:17.190598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.190576    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="609e08ae-ddf0-43a0-8854-110ccdcb2f02" containerName="revoke-keys"
Oct 31 02:05:17.190768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.190755    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="997cb194-76f1-4567-b54c-bd8f8f690bcb" containerName="payload-processing"
Oct 31 02:05:17.306066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.299638    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.353614 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.353581    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bb5c51\""
Oct 31 02:05:17.354128 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.354106    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"payload-processing-external-model-e2e--43356fff-dockercfg-qvkjn\""
Oct 31 02:05:17.354661 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.354641    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"agc-cred-e107317a3c62\""
Oct 31 02:05:17.354900 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.354884    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"kube-root-ca.crt\""
Oct 31 02:05:17.355156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.355140    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"routing-overlay\""
Oct 31 02:05:17.355361 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.355344    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"openshift-service-ca.crt\""
Oct 31 02:05:17.417066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.404234    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"]
Oct 31 02:05:17.419836 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.419748    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.458245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.458209    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.458730 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.458698    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jlwz\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-kube-api-access-7jlwz\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.464066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.459020    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.593066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.590053    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.593066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.590268    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.593066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.590403    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.593066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.590627    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jlwz\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-kube-api-access-7jlwz\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.607063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.606218    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.626804 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.626313    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.662603 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.662557    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.695407 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.678726    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jlwz\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-kube-api-access-7jlwz\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.752064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.726021    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw" podUID="5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" containerName="payload-processing" containerID="cri-o://8b054951b48170e832bd5fdea89514449cf6cca5b6ff0d338fea2c2d6c8730da" gracePeriod=25
Oct 31 02:05:17.844657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.844375    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:17.855747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.855238    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:17.906070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.904419    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:17.940837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:17.940798    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:18.034496 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.034392    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:18.049324 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.049275    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:18.064873 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.057386    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:18.121328 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.119353    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:18.214206 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.214165    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:18.235064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.231142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:18.468856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.468736    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:05:18.693829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.693786    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:18.889434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.889359    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:05:18.913112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.911985    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:05:18.913112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.912433    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:05:18.950674 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:18.950512    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5b52f6_10b6_488c_8fdb_b5f4c31d766f.slice/crio-conmon-8b054951b48170e832bd5fdea89514449cf6cca5b6ff0d338fea2c2d6c8730da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5b52f6_10b6_488c_8fdb_b5f4c31d766f.slice/crio-8b054951b48170e832bd5fdea89514449cf6cca5b6ff0d338fea2c2d6c8730da.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 02:05:18.963412 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:18.962923    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:05:19.078860 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.078824    5293 generic.go:356] "Generic (PLEG): container finished" podID="5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" containerID="8b054951b48170e832bd5fdea89514449cf6cca5b6ff0d338fea2c2d6c8730da" exitCode=0
Oct 31 02:05:19.079136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.078857    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw" event={"ID":"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f","Type":"ContainerDied","Data":"8b054951b48170e832bd5fdea89514449cf6cca5b6ff0d338fea2c2d6c8730da"}
Oct 31 02:05:19.397481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.397435    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"]
Oct 31 02:05:19.465200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.465170    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:05:19.505915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.504858    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"]
Oct 31 02:05:19.507115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.506288    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" containerName="payload-processing"
Oct 31 02:05:19.507115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.506311    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" containerName="payload-processing"
Oct 31 02:05:19.507115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.506604    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" containerName="payload-processing"
Oct 31 02:05:19.514881 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.513804    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.519967 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.519943    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-e2e--1b1bd4a9-dockercfg-7r8fh\""
Oct 31 02:05:19.520248 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.520229    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"kube-root-ca.crt\""
Oct 31 02:05:19.520377 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.520165    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"routing-overlay\""
Oct 31 02:05:19.520451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.520183    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-28e4f3\""
Oct 31 02:05:19.520527 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.520210    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"agc-cred-290407c1d817\""
Oct 31 02:05:19.520588 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.520131    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"openshift-service-ca.crt\""
Oct 31 02:05:19.528280 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.528257    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"]
Oct 31 02:05:19.568832 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.568803    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-routing-overlay-volume\") pod \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") "
Oct 31 02:05:19.568977 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.568857    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-provider-credentials\") pod \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") "
Oct 31 02:05:19.568977 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.568896    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-plugins-config-volume\") pod \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") "
Oct 31 02:05:19.568977 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.568957    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-kube-api-access-kh9kg\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-kube-api-access-kh9kg\") pod \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\" (UID: \"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f\") "
Oct 31 02:05:19.569348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.569317    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-routing-overlay-volume" pod "5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" (UID: "5e5b52f6-10b6-488c-8fdb-b5f4c31d766f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:19.570372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.570225    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:19.570476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.570414    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-plugins-config-volume" pod "5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" (UID: "5e5b52f6-10b6-488c-8fdb-b5f4c31d766f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:19.574149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.574127    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-provider-credentials" pod "5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" (UID: "5e5b52f6-10b6-488c-8fdb-b5f4c31d766f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:19.574859 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.574837    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-kube-api-access-kh9kg" pod "5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" (UID: "5e5b52f6-10b6-488c-8fdb-b5f4c31d766f"). InnerVolumeSpecName "kube-api-access-kh9kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:19.703599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.690289    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz8l4\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-kube-api-access-bz8l4\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.703599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.690467    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.703599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.690556    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.703599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.690590    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.703599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.690720    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:19.703599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.690738    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:19.703599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.690752    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh9kg\" (UniqueName: \"kubernetes.io/projected/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f-kube-api-access-kh9kg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:19.799753 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.799712    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz8l4\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-kube-api-access-bz8l4\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.814368 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.814331    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.814676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.814657    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.814816 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.814799    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.816063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.816010    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.817172 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.816757    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:19.817313 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:19.816851    5293 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:19.817409 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:19.817400    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:19.817539 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:19.817530    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials podName:c2f7287f-3be4-4639-811f-6a37a5398625 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:20.317509884 +0000 UTC m=+2678.266158266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl" (UID: "c2f7287f-3be4-4639-811f-6a37a5398625") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:19.860628 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:19.860406    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz8l4\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-kube-api-access-bz8l4\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:20.223090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.222462    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"
Oct 31 02:05:20.247904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.247846    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r" podStartSLOduration=4.247826812 podStartE2EDuration="4.247826812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:20.238423578 +0000 UTC m=+2678.187071971" watchObservedRunningTime="2026-10-31 02:05:20.247826812 +0000 UTC m=+2678.196475205"
Oct 31 02:05:20.290022 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.289984    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r" event={"ID":"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce","Type":"ContainerStarted","Data":"c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c"}
Oct 31 02:05:20.407268 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.407234    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:20.407586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.407561    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r" event={"ID":"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce","Type":"ContainerStarted","Data":"ee82bd37a4740677df5781630573d1788f85b772c862ad1ea12adbba6ec5c640"}
Oct 31 02:05:20.407757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.407737    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw" event={"ID":"5e5b52f6-10b6-488c-8fdb-b5f4c31d766f","Type":"ContainerDied","Data":"1fcbe0b7aa8d4ea9311a396beafc5b119cb2b641ebaaeeca98ee9b6542b657b3"}
Oct 31 02:05:20.407910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.407894    5293 scope.go:122] "RemoveContainer" containerID="8b054951b48170e832bd5fdea89514449cf6cca5b6ff0d338fea2c2d6c8730da"
Oct 31 02:05:20.408302 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.361654    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:20.532107 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:20.361749    5293 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:20.532107 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:20.527798    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:20.532107 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:20.527877    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials podName:c2f7287f-3be4-4639-811f-6a37a5398625 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:21.527853312 +0000 UTC m=+2679.476501700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl" (UID: "c2f7287f-3be4-4639-811f-6a37a5398625") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:20.547102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.532681    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"]
Oct 31 02:05:20.626504 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:20.622755    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl" podUID="c2f7287f-3be4-4639-811f-6a37a5398625"
Oct 31 02:05:20.654128 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.642685    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:05:20.676779 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.676740    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:20.798120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.772850    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"]
Oct 31 02:05:20.819210 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.817458    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b88hzjw"]
Oct 31 02:05:20.872065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.871332    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:05:20.923073 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.919369    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:20.980986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.980928    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:05:20.982702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:20.981364    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:21.079379 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.079189    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:21.089089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.087138    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:21.134055 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.133991    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:05:21.136234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.136197    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:05:21.166741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.166707    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"]
Oct 31 02:05:21.272383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.272280    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:05:21.361092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.350138    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:21.453874 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.453820    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5762d9-kkhq6"]
Oct 31 02:05:21.526065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.509204    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:21.559566 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.559152    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:21.635156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.618594    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:21.655280 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:21.632215    5293 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:21.667062 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:21.663147    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:21.667062 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:21.663244    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials podName:c2f7287f-3be4-4639-811f-6a37a5398625 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:23.663221402 +0000 UTC m=+2681.611869792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl" (UID: "c2f7287f-3be4-4639-811f-6a37a5398625") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:21.667062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.659155    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:21.667062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.663917    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:05:21.715141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.711136    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:21.715141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.711721    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:05:21.759782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.740761    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-plugins-config-volume\") pod \"c2f7287f-3be4-4639-811f-6a37a5398625\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") "
Oct 31 02:05:21.759782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.740818    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-routing-overlay-volume\") pod \"c2f7287f-3be4-4639-811f-6a37a5398625\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") "
Oct 31 02:05:21.759782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.741081    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-kube-api-access-bz8l4\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-kube-api-access-bz8l4\") pod \"c2f7287f-3be4-4639-811f-6a37a5398625\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") "
Oct 31 02:05:21.827422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.827386    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-routing-overlay-volume" pod "c2f7287f-3be4-4639-811f-6a37a5398625" (UID: "c2f7287f-3be4-4639-811f-6a37a5398625"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:21.839330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.839290    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-plugins-config-volume" pod "c2f7287f-3be4-4639-811f-6a37a5398625" (UID: "c2f7287f-3be4-4639-811f-6a37a5398625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:21.878067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.876118    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:21.909240 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.878690    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:05:21.972989 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.898278    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-kube-api-access-bz8l4" pod "c2f7287f-3be4-4639-811f-6a37a5398625" (UID: "c2f7287f-3be4-4639-811f-6a37a5398625"). InnerVolumeSpecName "kube-api-access-bz8l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:22.011513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:21.999947    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:22.022235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.022189    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:05:22.061072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.042130    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:22.061072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.042580    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:05:22.061072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.043283    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:05:22.061072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.043369    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:05:22.224192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.117431    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:05:22.224192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.118612    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:05:22.304670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.304628    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:05:22.359289 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.359226    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:22.396817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.396784    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz8l4\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-kube-api-access-bz8l4\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:22.397264 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.397235    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:22.397565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.397539    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7287f-3be4-4639-811f-6a37a5398625-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:22.600297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.600194    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:05:22.695091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.603985    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:23.753393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:22.816493    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:23.853519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:23.002280    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:23.853796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:23.002410    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:23.878739 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:23.128889    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:23.880556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:23.131419    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:24.005332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:23.342500    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:05:24.006575 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:23.396320    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:05:24.007088 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:23.396810    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:24.099219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.049118    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl\" (UID: \"c2f7287f-3be4-4639-811f-6a37a5398625\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:24.277114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.151803    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:05:24.277114 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:24.193387    5293 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:24.277114 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:24.272663    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:24.277114 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:24.272735    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials podName:c2f7287f-3be4-4639-811f-6a37a5398625 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:28.272703242 +0000 UTC m=+2686.221351628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl" (UID: "c2f7287f-3be4-4639-811f-6a37a5398625") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:24.391250 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.375477    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"
Oct 31 02:05:24.395080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.394905    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5b52f6-10b6-488c-8fdb-b5f4c31d766f" path="/var/lib/kubelet/pods/5e5b52f6-10b6-488c-8fdb-b5f4c31d766f/volumes"
Oct 31 02:05:24.397663 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.397642    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f05d658-ef99-4344-b7a0-1d691c708cca" path="/var/lib/kubelet/pods/9f05d658-ef99-4344-b7a0-1d691c708cca/volumes"
Oct 31 02:05:24.404264 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:24.404241    5293 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.179s"
Oct 31 02:05:24.452294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.452266    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:05:24.452632 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.452613    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:05:24.452801 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.452786    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:05:24.452928 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.452917    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.453058    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460347    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460385    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460414    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460443    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460472    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460507    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460548    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460576    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460599    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:05:24.462155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.460625    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:05:24.468003 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.467114    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:05:24.492064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.488451    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"]
Oct 31 02:05:24.522062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.522015    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"]
Oct 31 02:05:24.522263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.522203    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.523148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.523127    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-7d867dnwl"]
Oct 31 02:05:24.531992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.531926    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"]
Oct 31 02:05:24.532675 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.532655    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"openshift-service-ca.crt\""
Oct 31 02:05:24.533574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.533555    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"e2e-cfl-cec4cc-provider-cred\""
Oct 31 02:05:24.533815 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.533799    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-e2e--1b1bd4a9-dockercfg-7r8fh\""
Oct 31 02:05:24.534528 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.534508    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"kube-root-ca.crt\""
Oct 31 02:05:24.534719 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.534704    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-28e4f3\""
Oct 31 02:05:24.534915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.534902    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"routing-overlay\""
Oct 31 02:05:24.570108 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:24.570063    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f7287f_3be4_4639_811f_6a37a5398625.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 02:05:24.672845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.672807    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wqg\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-kube-api-access-h8wqg\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.673025 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.672988    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.673103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.673090    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.673260 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.673232    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.673606 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.673582    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7287f-3be4-4639-811f-6a37a5398625-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:24.775564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.775525    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wqg\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-kube-api-access-h8wqg\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.776098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.775670    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.776098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.775777    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.776098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.775839    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.776798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.776766    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.777127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.776782    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.779018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.778994    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.789174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.789117    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wqg\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-kube-api-access-h8wqg\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:24.864098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:24.859256    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:25.436587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:25.436559    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"]
Oct 31 02:05:26.178830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.178796    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f7287f-3be4-4639-811f-6a37a5398625" path="/var/lib/kubelet/pods/c2f7287f-3be4-4639-811f-6a37a5398625/volumes"
Oct 31 02:05:26.293018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.292978    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5" event={"ID":"10e20f0e-1fef-45d0-9585-409d7ef385c6","Type":"ContainerStarted","Data":"2e53b7434b82a717f9ff727a4d364073c8335ab96a418603ea02e0c43dc2fcd1"}
Oct 31 02:05:26.293271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.293095    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5" event={"ID":"10e20f0e-1fef-45d0-9585-409d7ef385c6","Type":"ContainerStarted","Data":"cc839a8d1c1225d4f4ac2d97816c8786574ad8cb79f989bd78d857cceaa932c1"}
Oct 31 02:05:26.293271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.293134    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:26.315910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.315864    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5" podStartSLOduration=6.315850589 podStartE2EDuration="6.315850589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:26.310658941 +0000 UTC m=+2684.259307332" watchObservedRunningTime="2026-10-31 02:05:26.315850589 +0000 UTC m=+2684.264498980"
Oct 31 02:05:26.520343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.520256    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"]
Oct 31 02:05:26.526408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.526386    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.529453 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.529429    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:05:26.529655 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.529513    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:05:26.536180 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.536154    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"]
Oct 31 02:05:26.607859 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.607821    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.608101 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.608083    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4pw\" (UniqueName: \"kubernetes.io/projected/7a5c56d2-712a-48f2-97f0-7fdabd381adf-kube-api-access-hh4pw\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.608275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.608261    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.709511 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.709477    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.709691 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.709591    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.709751 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.709739    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4pw\" (UniqueName: \"kubernetes.io/projected/7a5c56d2-712a-48f2-97f0-7fdabd381adf-kube-api-access-hh4pw\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.710372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.710344    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.712203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.712182    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.721922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.721896    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4pw\" (UniqueName: \"kubernetes.io/projected/7a5c56d2-712a-48f2-97f0-7fdabd381adf-kube-api-access-hh4pw\") pod \"maas-api-revoke-sub-6b89af60-qtrpq\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:26.848888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:26.848809    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:27.348734 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:27.348705    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"]
Oct 31 02:05:28.327971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:28.327870    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq" event={"ID":"7a5c56d2-712a-48f2-97f0-7fdabd381adf","Type":"ContainerStarted","Data":"47b12752f90c633044d5d7a16736d773d00ba3c274bed1f7e00dbb36e0d6d344"}
Oct 31 02:05:28.328213 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:28.328006    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq" event={"ID":"7a5c56d2-712a-48f2-97f0-7fdabd381adf","Type":"ContainerStarted","Data":"5ec05b337a75a02bf60bde55171d586586511efabd450ff413538e743588d4f2"}
Oct 31 02:05:28.432802 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:28.432737    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq" podStartSLOduration=2.432716558 podStartE2EDuration="2.432716558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:28.36555495 +0000 UTC m=+2686.314203378" watchObservedRunningTime="2026-10-31 02:05:28.432716558 +0000 UTC m=+2686.381364950"
Oct 31 02:05:29.342735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:29.342698    5293 generic.go:356] "Generic (PLEG): container finished" podID="7a5c56d2-712a-48f2-97f0-7fdabd381adf" containerID="47b12752f90c633044d5d7a16736d773d00ba3c274bed1f7e00dbb36e0d6d344" exitCode=0
Oct 31 02:05:29.342972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:29.342788    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq" event={"ID":"7a5c56d2-712a-48f2-97f0-7fdabd381adf","Type":"ContainerDied","Data":"47b12752f90c633044d5d7a16736d773d00ba3c274bed1f7e00dbb36e0d6d344"}
Oct 31 02:05:30.035299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.035260    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"]
Oct 31 02:05:30.039945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.039920    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa729f0f-k7hcc"]
Oct 31 02:05:30.180000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.179966    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc05133-0802-40d1-a375-5c05293962df" path="/var/lib/kubelet/pods/0cc05133-0802-40d1-a375-5c05293962df/volumes"
Oct 31 02:05:30.669828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.669759    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:30.758971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.758935    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5c56d2-712a-48f2-97f0-7fdabd381adf-kube-api-access-hh4pw\" (UniqueName: \"kubernetes.io/projected/7a5c56d2-712a-48f2-97f0-7fdabd381adf-kube-api-access-hh4pw\") pod \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") "
Oct 31 02:05:30.759359 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.759338    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-openshift-ca\") pod \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") "
Oct 31 02:05:30.759489 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.759453    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-xks-ca\") pod \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\" (UID: \"7a5c56d2-712a-48f2-97f0-7fdabd381adf\") "
Oct 31 02:05:30.759724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.759696    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-openshift-ca" pod "7a5c56d2-712a-48f2-97f0-7fdabd381adf" (UID: "7a5c56d2-712a-48f2-97f0-7fdabd381adf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:30.761192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.761161    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:30.761776 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.761742    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5c56d2-712a-48f2-97f0-7fdabd381adf-kube-api-access-hh4pw" pod "7a5c56d2-712a-48f2-97f0-7fdabd381adf" (UID: "7a5c56d2-712a-48f2-97f0-7fdabd381adf"). InnerVolumeSpecName "kube-api-access-hh4pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:30.761878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.761807    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-xks-ca" pod "7a5c56d2-712a-48f2-97f0-7fdabd381adf" (UID: "7a5c56d2-712a-48f2-97f0-7fdabd381adf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:05:30.862992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.862955    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5c56d2-712a-48f2-97f0-7fdabd381adf-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:30.862992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:30.862991    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh4pw\" (UniqueName: \"kubernetes.io/projected/7a5c56d2-712a-48f2-97f0-7fdabd381adf-kube-api-access-hh4pw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.394819 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:31.394784    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq" event={"ID":"7a5c56d2-712a-48f2-97f0-7fdabd381adf","Type":"ContainerDied","Data":"5ec05b337a75a02bf60bde55171d586586511efabd450ff413538e743588d4f2"}
Oct 31 02:05:31.394819 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:31.394812    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"
Oct 31 02:05:31.395396 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:31.394831    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ec05b337a75a02bf60bde55171d586586511efabd450ff413538e743588d4f2"
Oct 31 02:05:37.307011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:37.306982    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:05:38.304068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.295101    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"]
Oct 31 02:05:38.304068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.297016    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7a5c56d2-712a-48f2-97f0-7fdabd381adf" containerName="revoke-keys"
Oct 31 02:05:38.304068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.297061    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5c56d2-712a-48f2-97f0-7fdabd381adf" containerName="revoke-keys"
Oct 31 02:05:38.304068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.297380    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5c56d2-712a-48f2-97f0-7fdabd381adf" containerName="revoke-keys"
Oct 31 02:05:38.514735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.514699    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.534810 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.534771    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:05:38.535493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.535471    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:05:38.783064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.780157    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.783064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.780211    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkd4z\" (UniqueName: \"kubernetes.io/projected/a33e958f-e08e-4cbe-920d-188ff8a89b23-kube-api-access-zkd4z\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.783064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.780352    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.783064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.780572    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"]
Oct 31 02:05:38.882447 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.882415    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.882447 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.882448    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkd4z\" (UniqueName: \"kubernetes.io/projected/a33e958f-e08e-4cbe-920d-188ff8a89b23-kube-api-access-zkd4z\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.882657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.882549    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.886415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.886390    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.887651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.887630    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:38.894590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:38.894566    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkd4z\" (UniqueName: \"kubernetes.io/projected/a33e958f-e08e-4cbe-920d-188ff8a89b23-kube-api-access-zkd4z\") pod \"maas-api-revoke-sub-895d3754-vsx97\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:39.124957 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:39.124861    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:39.577235 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:05:39.577211    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33e958f_e08e_4cbe_920d_188ff8a89b23.slice/crio-67d59f47150a61204c86e0b8c65bb228de98d3564914fd280dc5da1f0d6bcae1 WatchSource:0}: Error finding container 67d59f47150a61204c86e0b8c65bb228de98d3564914fd280dc5da1f0d6bcae1: Status 404 returned error can't find the container with id 67d59f47150a61204c86e0b8c65bb228de98d3564914fd280dc5da1f0d6bcae1
Oct 31 02:05:39.578293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:39.578271    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"]
Oct 31 02:05:39.808375 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:39.808337    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97" event={"ID":"a33e958f-e08e-4cbe-920d-188ff8a89b23","Type":"ContainerStarted","Data":"67d59f47150a61204c86e0b8c65bb228de98d3564914fd280dc5da1f0d6bcae1"}
Oct 31 02:05:40.820783 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:40.820750    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97" event={"ID":"a33e958f-e08e-4cbe-920d-188ff8a89b23","Type":"ContainerStarted","Data":"54132afb1faad0ad2b599b6a48e35a8c6051815531786ac44df03d4f816e0b49"}
Oct 31 02:05:40.843438 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:40.843384    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97" podStartSLOduration=2.84336918 podStartE2EDuration="2.84336918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:40.837934258 +0000 UTC m=+2698.786582650" watchObservedRunningTime="2026-10-31 02:05:40.84336918 +0000 UTC m=+2698.792017571"
Oct 31 02:05:41.786093 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:41.786014    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:05:41.835226 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:41.834719    5293 generic.go:356] "Generic (PLEG): container finished" podID="a33e958f-e08e-4cbe-920d-188ff8a89b23" containerID="54132afb1faad0ad2b599b6a48e35a8c6051815531786ac44df03d4f816e0b49" exitCode=0
Oct 31 02:05:41.835226 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:41.835022    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97" event={"ID":"a33e958f-e08e-4cbe-920d-188ff8a89b23","Type":"ContainerDied","Data":"54132afb1faad0ad2b599b6a48e35a8c6051815531786ac44df03d4f816e0b49"}
Oct 31 02:05:43.079823 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.076906    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:43.256394 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.256355    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-xks-ca\") pod \"a33e958f-e08e-4cbe-920d-188ff8a89b23\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") "
Oct 31 02:05:43.256576 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.256497    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33e958f-e08e-4cbe-920d-188ff8a89b23-kube-api-access-zkd4z\" (UniqueName: \"kubernetes.io/projected/a33e958f-e08e-4cbe-920d-188ff8a89b23-kube-api-access-zkd4z\") pod \"a33e958f-e08e-4cbe-920d-188ff8a89b23\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") "
Oct 31 02:05:43.256668 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.256621    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-openshift-ca\") pod \"a33e958f-e08e-4cbe-920d-188ff8a89b23\" (UID: \"a33e958f-e08e-4cbe-920d-188ff8a89b23\") "
Oct 31 02:05:43.257000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.256961    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-openshift-ca" pod "a33e958f-e08e-4cbe-920d-188ff8a89b23" (UID: "a33e958f-e08e-4cbe-920d-188ff8a89b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:43.257873 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.257841    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:43.258981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.258950    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-xks-ca" pod "a33e958f-e08e-4cbe-920d-188ff8a89b23" (UID: "a33e958f-e08e-4cbe-920d-188ff8a89b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:05:43.259186 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.259151    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33e958f-e08e-4cbe-920d-188ff8a89b23-kube-api-access-zkd4z" pod "a33e958f-e08e-4cbe-920d-188ff8a89b23" (UID: "a33e958f-e08e-4cbe-920d-188ff8a89b23"). InnerVolumeSpecName "kube-api-access-zkd4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:43.364183 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.364139    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33e958f-e08e-4cbe-920d-188ff8a89b23-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:43.376885 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:43.376843    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkd4z\" (UniqueName: \"kubernetes.io/projected/a33e958f-e08e-4cbe-920d-188ff8a89b23-kube-api-access-zkd4z\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:44.251851 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:44.251817    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"
Oct 31 02:05:44.304444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:44.303567    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97" event={"ID":"a33e958f-e08e-4cbe-920d-188ff8a89b23","Type":"ContainerDied","Data":"67d59f47150a61204c86e0b8c65bb228de98d3564914fd280dc5da1f0d6bcae1"}
Oct 31 02:05:44.304444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:44.303634    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d59f47150a61204c86e0b8c65bb228de98d3564914fd280dc5da1f0d6bcae1"
Oct 31 02:05:44.384684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:44.378286    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:44.391593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:44.391157    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:44.485330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:44.483327    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:44.660027 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:44.659987    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:44.888305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:44.887317    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:45.011907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.010330    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:45.012209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.012181    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:05:45.012631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.012334    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:05:45.142064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.133343    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:45.142064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.133728    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 02:05:45.226373 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.207156    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:45.226373 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.208285    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 02:05:45.332117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.326250    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:05:45.396069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.391580    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:05:45.396069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.392250    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:05:45.400066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.396345    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:45.400066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.396851    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:05:45.499064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.497683    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:05:45.713223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.713188    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:05:45.739584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.739553    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:05:45.903994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.889118    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:05:45.903994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.889595    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:05:45.928741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.914133    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:05:45.928741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:45.918788    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:05:46.010869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:46.010772    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:46.058317 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:46.058234    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:05:46.077782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:46.077757    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:05:46.132862 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:46.132820    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:05:49.528626 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:49.528589    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"]
Oct 31 02:05:54.529603 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:54.529501    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn" podUID="b1c5673b-5d4c-4518-9088-d0f9304638a5" containerName="payload-processing" containerID="cri-o://66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c" gracePeriod=25
Oct 31 02:05:54.930588 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:54.930552    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:55.022542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.022505    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-kube-api-access-kz8zh\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-kube-api-access-kz8zh\") pod \"b1c5673b-5d4c-4518-9088-d0f9304638a5\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") "
Oct 31 02:05:55.022744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.022647    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-plugins-config-volume\") pod \"b1c5673b-5d4c-4518-9088-d0f9304638a5\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") "
Oct 31 02:05:55.022818 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.022779    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-routing-overlay-volume\") pod \"b1c5673b-5d4c-4518-9088-d0f9304638a5\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") "
Oct 31 02:05:55.022880 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.022864    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-provider-credentials\") pod \"b1c5673b-5d4c-4518-9088-d0f9304638a5\" (UID: \"b1c5673b-5d4c-4518-9088-d0f9304638a5\") "
Oct 31 02:05:55.023094 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.023068    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-routing-overlay-volume" pod "b1c5673b-5d4c-4518-9088-d0f9304638a5" (UID: "b1c5673b-5d4c-4518-9088-d0f9304638a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:55.023174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.023115    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-plugins-config-volume" pod "b1c5673b-5d4c-4518-9088-d0f9304638a5" (UID: "b1c5673b-5d4c-4518-9088-d0f9304638a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:55.024030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.024003    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:55.024168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.024092    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5673b-5d4c-4518-9088-d0f9304638a5-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:55.025486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.025465    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-provider-credentials" pod "b1c5673b-5d4c-4518-9088-d0f9304638a5" (UID: "b1c5673b-5d4c-4518-9088-d0f9304638a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:55.025590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.025506    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-kube-api-access-kz8zh" pod "b1c5673b-5d4c-4518-9088-d0f9304638a5" (UID: "b1c5673b-5d4c-4518-9088-d0f9304638a5"). InnerVolumeSpecName "kube-api-access-kz8zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:55.125675 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.125602    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:55.125675 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.125631    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz8zh\" (UniqueName: \"kubernetes.io/projected/b1c5673b-5d4c-4518-9088-d0f9304638a5-kube-api-access-kz8zh\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:55.642296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.642261    5293 generic.go:356] "Generic (PLEG): container finished" podID="b1c5673b-5d4c-4518-9088-d0f9304638a5" containerID="66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c" exitCode=0
Oct 31 02:05:55.642815 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.642317    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"
Oct 31 02:05:55.642815 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.642330    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn" event={"ID":"b1c5673b-5d4c-4518-9088-d0f9304638a5","Type":"ContainerDied","Data":"66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c"}
Oct 31 02:05:55.642815 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.642374    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn" event={"ID":"b1c5673b-5d4c-4518-9088-d0f9304638a5","Type":"ContainerDied","Data":"3a37606926a5f755347d7169dc1849941f76036de64553a8a707e61f823ff88d"}
Oct 31 02:05:55.642815 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.642395    5293 scope.go:122] "RemoveContainer" containerID="66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c"
Oct 31 02:05:55.656544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.656522    5293 scope.go:122] "RemoveContainer" containerID="66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c"
Oct 31 02:05:55.656935 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:05:55.656900    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c\": container with ID starting with 66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c not found: ID does not exist" containerID="66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c"
Oct 31 02:05:55.657108 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.656945    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c"} err="failed to get container status \"66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c\": rpc error: code = NotFound desc = could not find container \"66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c\": container with ID starting with 66fd3e2e6a841b27e5a396ebee776db38a6e7bb476bf55c235a2638bf9438f7c not found: ID does not exist"
Oct 31 02:05:55.671573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.671513    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"]
Oct 31 02:05:55.676539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:55.676516    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-78f664lqn"]
Oct 31 02:05:56.358547 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:05:56.355073    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c5673b-5d4c-4518-9088-d0f9304638a5" path="/var/lib/kubelet/pods/b1c5673b-5d4c-4518-9088-d0f9304638a5/volumes"
Oct 31 02:06:04.542440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:04.542393    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:06:05.450618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.450579    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"]
Oct 31 02:06:05.451747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.451727    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a33e958f-e08e-4cbe-920d-188ff8a89b23" containerName="revoke-keys"
Oct 31 02:06:05.451747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.451748    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33e958f-e08e-4cbe-920d-188ff8a89b23" containerName="revoke-keys"
Oct 31 02:06:05.451885 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.451776    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b1c5673b-5d4c-4518-9088-d0f9304638a5" containerName="payload-processing"
Oct 31 02:06:05.451885 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.451781    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c5673b-5d4c-4518-9088-d0f9304638a5" containerName="payload-processing"
Oct 31 02:06:05.451964 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.451933    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33e958f-e08e-4cbe-920d-188ff8a89b23" containerName="revoke-keys"
Oct 31 02:06:05.451964 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.451942    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c5673b-5d4c-4518-9088-d0f9304638a5" containerName="payload-processing"
Oct 31 02:06:05.462568 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.462530    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.467410 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.467387    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qsc4d\""
Oct 31 02:06:05.467849 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.467830    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 02:06:05.467995 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.467972    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 02:06:05.468163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.468144    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6e1bafd9570a\""
Oct 31 02:06:05.468340 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.468324    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 02:06:05.468459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.467896    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 02:06:05.472270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.470088    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"]
Oct 31 02:06:05.551502 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.551460    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-plugins-config-volume\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.552120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.552093    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hz2\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-kube-api-access-f2hz2\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.552428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.552401    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-routing-overlay-volume\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.552625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.552490    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-provider-credentials\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.686072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.658852    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hz2\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-kube-api-access-f2hz2\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.686072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.659112    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-routing-overlay-volume\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.686072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.659218    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-provider-credentials\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.686072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.659402    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-plugins-config-volume\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.686072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.660874    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-plugins-config-volume\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.686072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.662155    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-routing-overlay-volume\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.727487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.714486    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-provider-credentials\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:05.826065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:05.800385    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hz2\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-kube-api-access-f2hz2\") pod \"payload-processing-external-model-5c8bffbccc-lfg9b\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") " pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:06.094639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:06.094550    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:07.037667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:07.037629    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"]
Oct 31 02:06:07.094614 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:07.077426    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9165178-kzjh4"]
Oct 31 02:06:07.159594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:07.159565    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"]
Oct 31 02:06:07.965912 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:07.965874    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b" event={"ID":"f4c5fe57-9b80-4679-a263-2515de1eef94","Type":"ContainerStarted","Data":"31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7"}
Oct 31 02:06:07.966147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:07.965929    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b" event={"ID":"f4c5fe57-9b80-4679-a263-2515de1eef94","Type":"ContainerStarted","Data":"85366bdd894dd726bc61e093e1bd41cdb80aafb54baa79d5cef88b7e70ad3eaf"}
Oct 31 02:06:07.966147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:07.965970    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:07.987848 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:07.987765    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b" podStartSLOduration=2.987751143 podStartE2EDuration="2.987751143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:06:07.984641759 +0000 UTC m=+2725.933290150" watchObservedRunningTime="2026-10-31 02:06:07.987751143 +0000 UTC m=+2725.936399531"
Oct 31 02:06:08.222266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:08.222183    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc9a141-ddb1-4b88-9561-d8b504634594" path="/var/lib/kubelet/pods/8cc9a141-ddb1-4b88-9561-d8b504634594/volumes"
Oct 31 02:06:11.782115 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:06:11.782076    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:06:17.026223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:17.026187    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"]
Oct 31 02:06:17.031404 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:17.031372    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d7c4745-sgrxq"]
Oct 31 02:06:18.178079 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:18.178024    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f460eb08-fec3-45ea-a6e1-4ee09babab4e" path="/var/lib/kubelet/pods/f460eb08-fec3-45ea-a6e1-4ee09babab4e/volumes"
Oct 31 02:06:18.978738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:18.978712    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:06:41.783407 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:06:41.783372    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:06:45.981006 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:45.980972    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:06:48.085724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.085680    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:06:48.110160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.109119    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:06:48.117365 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.117330    5293 scope.go:122] "RemoveContainer" containerID="6110044fb2ef081435b22c24e4bd4bb259de99e72fa912123f93f6e99a59a64c"
Oct 31 02:06:48.145429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.141148    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:06:48.170370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.170339    5293 scope.go:122] "RemoveContainer" containerID="7ae5ba175adb8bcc8f31668ec4a303107761a61386e0f65c60bc2f0d890c5185"
Oct 31 02:06:48.266710 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.265436    5293 scope.go:122] "RemoveContainer" containerID="8cb904b5a68dbd206ee7d28f1cc6fe6c935c532c93d34a004354fed3498051f3"
Oct 31 02:06:48.274558 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.274523    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:06:48.298787 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.298766    5293 scope.go:122] "RemoveContainer" containerID="05609646e758487db92930ac86d50aa69411d81a4285704e914af1eb03ee4735"
Oct 31 02:06:48.311276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:48.311249    5293 scope.go:122] "RemoveContainer" containerID="f1b223c35a9fa44a310d94c9641c0664f8d4028055d5aa1c214343908cad0b75"
Oct 31 02:06:49.505079 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.505003    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"]
Oct 31 02:06:49.512028 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.512004    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.516097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.515271    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:06:49.516097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.515778    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:06:49.516405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.516385    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"]
Oct 31 02:06:49.671676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.671642    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rf7\" (UniqueName: \"kubernetes.io/projected/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-kube-api-access-78rf7\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.671884 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.671801    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.672000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.671980    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.773385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.773292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rf7\" (UniqueName: \"kubernetes.io/projected/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-kube-api-access-78rf7\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.773588 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.773405    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.773588 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.773564    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.774225 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.774201    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.775998 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.775979    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.785097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.785071    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rf7\" (UniqueName: \"kubernetes.io/projected/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-kube-api-access-78rf7\") pod \"maas-api-revoke-sub-78a5c53b-mntwh\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:49.838741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:49.838710    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:50.253572 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:50.253541    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"]
Oct 31 02:06:50.821611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:50.821575    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh" event={"ID":"1f6540f6-e146-40ea-9227-3a9aa21bf5d2","Type":"ContainerStarted","Data":"c242158c1895bd20d79e214246396eb7b0c7c99e2dcb5ee9a7d4014916a25963"}
Oct 31 02:06:51.913270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:51.912819    5293 generic.go:356] "Generic (PLEG): container finished" podID="1f6540f6-e146-40ea-9227-3a9aa21bf5d2" containerID="4946f390a5ce951ba04d07175dfa2cfab4a873c60262d3967a17e5fb3a718f6a" exitCode=0
Oct 31 02:06:51.913270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:51.913078    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh" event={"ID":"1f6540f6-e146-40ea-9227-3a9aa21bf5d2","Type":"ContainerDied","Data":"4946f390a5ce951ba04d07175dfa2cfab4a873c60262d3967a17e5fb3a718f6a"}
Oct 31 02:06:53.273505 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.273479    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:53.410378 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.410346    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-kube-api-access-78rf7\" (UniqueName: \"kubernetes.io/projected/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-kube-api-access-78rf7\") pod \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") "
Oct 31 02:06:53.410673 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.410650    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-openshift-ca\") pod \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") "
Oct 31 02:06:53.410813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.410748    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-xks-ca\") pod \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\" (UID: \"1f6540f6-e146-40ea-9227-3a9aa21bf5d2\") "
Oct 31 02:06:53.411016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.410986    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-openshift-ca" pod "1f6540f6-e146-40ea-9227-3a9aa21bf5d2" (UID: "1f6540f6-e146-40ea-9227-3a9aa21bf5d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:53.411625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.411603    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.412873 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.412842    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-kube-api-access-78rf7" pod "1f6540f6-e146-40ea-9227-3a9aa21bf5d2" (UID: "1f6540f6-e146-40ea-9227-3a9aa21bf5d2"). InnerVolumeSpecName "kube-api-access-78rf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:53.412986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.412963    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-xks-ca" pod "1f6540f6-e146-40ea-9227-3a9aa21bf5d2" (UID: "1f6540f6-e146-40ea-9227-3a9aa21bf5d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:53.513481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.513400    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78rf7\" (UniqueName: \"kubernetes.io/projected/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-kube-api-access-78rf7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.513481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.513433    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6540f6-e146-40ea-9227-3a9aa21bf5d2-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.936325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.936282    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"
Oct 31 02:06:53.936493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.936298    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh" event={"ID":"1f6540f6-e146-40ea-9227-3a9aa21bf5d2","Type":"ContainerDied","Data":"c242158c1895bd20d79e214246396eb7b0c7c99e2dcb5ee9a7d4014916a25963"}
Oct 31 02:06:53.936493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:06:53.936422    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c242158c1895bd20d79e214246396eb7b0c7c99e2dcb5ee9a7d4014916a25963"
Oct 31 02:07:11.780845 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:07:11.780809    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:07:13.480063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:13.471821    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:07:13.927254 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:13.927172    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:07:14.039065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:14.029948    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:07:14.099066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:14.096404    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:07:14.130065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:14.129298    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:07:14.141361 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:14.141322    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:07:14.206075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:14.204400    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:07:14.278304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:14.278250    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:07:16.026621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.026584    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"]
Oct 31 02:07:16.027786 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.027761    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1f6540f6-e146-40ea-9227-3a9aa21bf5d2" containerName="revoke-keys"
Oct 31 02:07:16.027887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.027790    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6540f6-e146-40ea-9227-3a9aa21bf5d2" containerName="revoke-keys"
Oct 31 02:07:16.028013 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.027999    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6540f6-e146-40ea-9227-3a9aa21bf5d2" containerName="revoke-keys"
Oct 31 02:07:16.032103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.032084    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.035958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.035903    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:16.036217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.036192    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:07:16.043055 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.043009    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"]
Oct 31 02:07:16.185290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.185242    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zll9f\" (UniqueName: \"kubernetes.io/projected/3175419f-620d-4893-b501-8f146f42a5f0-kube-api-access-zll9f\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.185639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.185473    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.185639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.185632    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.287824 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.287732    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.288002 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.287901    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.288169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.288151    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zll9f\" (UniqueName: \"kubernetes.io/projected/3175419f-620d-4893-b501-8f146f42a5f0-kube-api-access-zll9f\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.288551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.288527    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.291273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.291239    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.302574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.302546    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zll9f\" (UniqueName: \"kubernetes.io/projected/3175419f-620d-4893-b501-8f146f42a5f0-kube-api-access-zll9f\") pod \"maas-api-revoke-sub-ae625aeb-l2899\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.352187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.352162    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:16.667773 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:07:16.667740    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3175419f_620d_4893_b501_8f146f42a5f0.slice/crio-d8feff44e3e318eb53b98602d14aee27eafdc3f74268311a8fbef7775dd3f094 WatchSource:0}: Error finding container d8feff44e3e318eb53b98602d14aee27eafdc3f74268311a8fbef7775dd3f094: Status 404 returned error can't find the container with id d8feff44e3e318eb53b98602d14aee27eafdc3f74268311a8fbef7775dd3f094
Oct 31 02:07:16.668428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.668401    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"]
Oct 31 02:07:16.859356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:16.859318    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899" event={"ID":"3175419f-620d-4893-b501-8f146f42a5f0","Type":"ContainerStarted","Data":"d8feff44e3e318eb53b98602d14aee27eafdc3f74268311a8fbef7775dd3f094"}
Oct 31 02:07:17.868764 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:17.868729    5293 generic.go:356] "Generic (PLEG): container finished" podID="3175419f-620d-4893-b501-8f146f42a5f0" containerID="66b0d10781ec47e00e632e37e7a1e0da5286847669010bb7fb67fbd38ef81f93" exitCode=0
Oct 31 02:07:17.869203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:17.868808    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899" event={"ID":"3175419f-620d-4893-b501-8f146f42a5f0","Type":"ContainerDied","Data":"66b0d10781ec47e00e632e37e7a1e0da5286847669010bb7fb67fbd38ef81f93"}
Oct 31 02:07:18.158803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:18.158766    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"]
Oct 31 02:07:19.091515 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.091482    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:19.171145 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.171114    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3175419f-620d-4893-b501-8f146f42a5f0-kube-api-access-zll9f\" (UniqueName: \"kubernetes.io/projected/3175419f-620d-4893-b501-8f146f42a5f0-kube-api-access-zll9f\") pod \"3175419f-620d-4893-b501-8f146f42a5f0\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") "
Oct 31 02:07:19.171486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.171462    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-xks-ca\") pod \"3175419f-620d-4893-b501-8f146f42a5f0\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") "
Oct 31 02:07:19.171619 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.171602    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-openshift-ca\") pod \"3175419f-620d-4893-b501-8f146f42a5f0\" (UID: \"3175419f-620d-4893-b501-8f146f42a5f0\") "
Oct 31 02:07:19.172086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.172016    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-openshift-ca" pod "3175419f-620d-4893-b501-8f146f42a5f0" (UID: "3175419f-620d-4893-b501-8f146f42a5f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:19.172482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.172459    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:19.174298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.174272    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-xks-ca" pod "3175419f-620d-4893-b501-8f146f42a5f0" (UID: "3175419f-620d-4893-b501-8f146f42a5f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:19.174432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.174320    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3175419f-620d-4893-b501-8f146f42a5f0-kube-api-access-zll9f" pod "3175419f-620d-4893-b501-8f146f42a5f0" (UID: "3175419f-620d-4893-b501-8f146f42a5f0"). InnerVolumeSpecName "kube-api-access-zll9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:19.328642 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.328605    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zll9f\" (UniqueName: \"kubernetes.io/projected/3175419f-620d-4893-b501-8f146f42a5f0-kube-api-access-zll9f\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:19.333198 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.333163    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175419f-620d-4893-b501-8f146f42a5f0-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:19.888054 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.888009    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899" event={"ID":"3175419f-620d-4893-b501-8f146f42a5f0","Type":"ContainerDied","Data":"d8feff44e3e318eb53b98602d14aee27eafdc3f74268311a8fbef7775dd3f094"}
Oct 31 02:07:19.888263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.888116    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8feff44e3e318eb53b98602d14aee27eafdc3f74268311a8fbef7775dd3f094"
Oct 31 02:07:19.888263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:19.888193    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"
Oct 31 02:07:23.074825 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.074744    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"]
Oct 31 02:07:23.075705 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.075688    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3175419f-620d-4893-b501-8f146f42a5f0" containerName="revoke-keys"
Oct 31 02:07:23.075747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.075707    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3175419f-620d-4893-b501-8f146f42a5f0" containerName="revoke-keys"
Oct 31 02:07:23.075850 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.075841    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3175419f-620d-4893-b501-8f146f42a5f0" containerName="revoke-keys"
Oct 31 02:07:23.080226 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.080205    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.083270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.083245    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:23.083407 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.083292    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:07:23.090164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.090138    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"]
Oct 31 02:07:23.160123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.160082    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r" podUID="9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" containerName="payload-processing" containerID="cri-o://c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c" gracePeriod=25
Oct 31 02:07:23.184138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.184105    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.184269 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.184171    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8tp\" (UniqueName: \"kubernetes.io/projected/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-kube-api-access-pp8tp\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.184321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.184287    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.285629 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.285600    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.285775 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.285660    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8tp\" (UniqueName: \"kubernetes.io/projected/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-kube-api-access-pp8tp\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.285865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.285789    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.286809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.286781    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.288333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.288313    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.297899 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.297875    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8tp\" (UniqueName: \"kubernetes.io/projected/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-kube-api-access-pp8tp\") pod \"maas-api-revoke-sub-5ac4f642-9b4mj\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.416931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.416611    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:23.497053 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.497011    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:07:23.546384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.545435    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"]
Oct 31 02:07:23.546860 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.546834    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" containerName="payload-processing"
Oct 31 02:07:23.546860 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.546860    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" containerName="payload-processing"
Oct 31 02:07:23.547112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.547064    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" containerName="payload-processing"
Oct 31 02:07:23.560348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.560291    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"]
Oct 31 02:07:23.560481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.560403    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.567333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.566872    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"kube-root-ca.crt\""
Oct 31 02:07:23.567333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.567058    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"openshift-service-ca.crt\""
Oct 31 02:07:23.567333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.567030    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-20538a\""
Oct 31 02:07:23.567333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.567128    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"agc-cred-c47ea10560ab\""
Oct 31 02:07:23.567333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.567170    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"routing-overlay\""
Oct 31 02:07:23.567333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.567230    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"payload-processing-external-model-e2e--968a6914-dockercfg-99978\""
Oct 31 02:07:23.591686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.591659    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-kube-api-access-7jlwz\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-kube-api-access-7jlwz\") pod \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") "
Oct 31 02:07:23.591872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.591789    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-routing-overlay-volume\") pod \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") "
Oct 31 02:07:23.591946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.591930    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-plugins-config-volume\") pod \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") "
Oct 31 02:07:23.592481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.592293    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-plugins-config-volume" pod "9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" (UID: "9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:23.592481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.592304    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-provider-credentials\") pod \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\" (UID: \"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce\") "
Oct 31 02:07:23.592481 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.592441    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-routing-overlay-volume" pod "9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" (UID: "9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:23.593322 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.593301    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:23.593415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.593328    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:23.594877 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.594829    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-kube-api-access-7jlwz" pod "9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" (UID: "9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce"). InnerVolumeSpecName "kube-api-access-7jlwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:23.596723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.596700    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-provider-credentials" pod "9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" (UID: "9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:23.725077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.721115    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.725077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.721440    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.725077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.721483    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.725077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.721570    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9k8\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-kube-api-access-ss9k8\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.725077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.721713    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:23.725077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.721734    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jlwz\" (UniqueName: \"kubernetes.io/projected/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce-kube-api-access-7jlwz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:23.811015 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.810603    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"]
Oct 31 02:07:23.822704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.822675    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.822795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.822719    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.822795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.822764    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9k8\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-kube-api-access-ss9k8\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.822922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.822877    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.823693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.823645    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.823836 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.823788    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.825955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.825929    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.835742 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.835716    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9k8\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-kube-api-access-ss9k8\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.881272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.881248    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:23.939296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.939260    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj" event={"ID":"de172b2b-f10a-48dc-bed8-4b3ff0788ffb","Type":"ContainerStarted","Data":"5c36f19a46cd912f15f2aeb351aa2193d2e935c61c473a398abe5f7fc43c333a"}
Oct 31 02:07:23.941135 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.941106    5293 generic.go:356] "Generic (PLEG): container finished" podID="9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" containerID="c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c" exitCode=0
Oct 31 02:07:23.941305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.941175    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"
Oct 31 02:07:23.941305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.941185    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r" event={"ID":"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce","Type":"ContainerDied","Data":"c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c"}
Oct 31 02:07:23.941305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.941243    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r" event={"ID":"9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce","Type":"ContainerDied","Data":"ee82bd37a4740677df5781630573d1788f85b772c862ad1ea12adbba6ec5c640"}
Oct 31 02:07:23.941305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.941274    5293 scope.go:122] "RemoveContainer" containerID="c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c"
Oct 31 02:07:23.953914 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.953896    5293 scope.go:122] "RemoveContainer" containerID="c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c"
Oct 31 02:07:23.954259 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:07:23.954234    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c\": container with ID starting with c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c not found: ID does not exist" containerID="c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c"
Oct 31 02:07:23.954334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.954267    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c"} err="failed to get container status \"c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c\": rpc error: code = NotFound desc = could not find container \"c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c\": container with ID starting with c8d4ef305cc24bed5e5d644319f2c289547ab015aa84a9bcc7982eac84610a1c not found: ID does not exist"
Oct 31 02:07:23.968994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.968968    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"]
Oct 31 02:07:23.975356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:23.975334    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8n97r"]
Oct 31 02:07:24.177416 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:24.177386    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce" path="/var/lib/kubelet/pods/9b5c55b2-15a4-4ae1-afa7-6cd8da6eddce/volumes"
Oct 31 02:07:24.278332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:24.278300    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"]
Oct 31 02:07:24.299480 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:07:24.299447    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a9a213_af7f_47c7_b31d_d021c33f2d2b.slice/crio-6a099926f6a4c4b272f2304ebe011e999f27a2cefc207c4196094ab4f21ffb60 WatchSource:0}: Error finding container 6a099926f6a4c4b272f2304ebe011e999f27a2cefc207c4196094ab4f21ffb60: Status 404 returned error can't find the container with id 6a099926f6a4c4b272f2304ebe011e999f27a2cefc207c4196094ab4f21ffb60
Oct 31 02:07:24.953167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:24.953132    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj" event={"ID":"de172b2b-f10a-48dc-bed8-4b3ff0788ffb","Type":"ContainerStarted","Data":"8e9ad5f3512c5a0095855aab51d3eb76706b01b8b5b8928e58464b8405bb7a9d"}
Oct 31 02:07:24.954873 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:24.954838    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" event={"ID":"93a9a213-af7f-47c7-b31d-d021c33f2d2b","Type":"ContainerStarted","Data":"8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96"}
Oct 31 02:07:24.955011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:24.954880    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" event={"ID":"93a9a213-af7f-47c7-b31d-d021c33f2d2b","Type":"ContainerStarted","Data":"6a099926f6a4c4b272f2304ebe011e999f27a2cefc207c4196094ab4f21ffb60"}
Oct 31 02:07:24.955011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:24.954977    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:07:24.973366 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:24.973322    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj" podStartSLOduration=1.973308852 podStartE2EDuration="1.973308852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:24.971008859 +0000 UTC m=+2802.919657251" watchObservedRunningTime="2026-10-31 02:07:24.973308852 +0000 UTC m=+2802.921957244"
Oct 31 02:07:24.993423 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:24.993380    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" podStartSLOduration=21.993367479 podStartE2EDuration="21.993367479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:24.990483637 +0000 UTC m=+2802.939132028" watchObservedRunningTime="2026-10-31 02:07:24.993367479 +0000 UTC m=+2802.942015902"
Oct 31 02:07:25.966971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:25.966936    5293 generic.go:356] "Generic (PLEG): container finished" podID="de172b2b-f10a-48dc-bed8-4b3ff0788ffb" containerID="8e9ad5f3512c5a0095855aab51d3eb76706b01b8b5b8928e58464b8405bb7a9d" exitCode=0
Oct 31 02:07:25.967414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:25.967019    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj" event={"ID":"de172b2b-f10a-48dc-bed8-4b3ff0788ffb","Type":"ContainerDied","Data":"8e9ad5f3512c5a0095855aab51d3eb76706b01b8b5b8928e58464b8405bb7a9d"}
Oct 31 02:07:27.171830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.171807    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:27.300534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.300447    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-openshift-ca\") pod \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") "
Oct 31 02:07:27.300694 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.300664    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-kube-api-access-pp8tp\" (UniqueName: \"kubernetes.io/projected/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-kube-api-access-pp8tp\") pod \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") "
Oct 31 02:07:27.300777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.300753    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-openshift-ca" pod "de172b2b-f10a-48dc-bed8-4b3ff0788ffb" (UID: "de172b2b-f10a-48dc-bed8-4b3ff0788ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:27.300863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.300847    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-xks-ca\") pod \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\" (UID: \"de172b2b-f10a-48dc-bed8-4b3ff0788ffb\") "
Oct 31 02:07:27.301719 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.301698    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.302963 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.302938    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-xks-ca" pod "de172b2b-f10a-48dc-bed8-4b3ff0788ffb" (UID: "de172b2b-f10a-48dc-bed8-4b3ff0788ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:27.303179 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.303161    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-kube-api-access-pp8tp" pod "de172b2b-f10a-48dc-bed8-4b3ff0788ffb" (UID: "de172b2b-f10a-48dc-bed8-4b3ff0788ffb"). InnerVolumeSpecName "kube-api-access-pp8tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:27.405007 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.404888    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8tp\" (UniqueName: \"kubernetes.io/projected/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-kube-api-access-pp8tp\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.405007 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.404923    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de172b2b-f10a-48dc-bed8-4b3ff0788ffb-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.986648 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.986614    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"
Oct 31 02:07:27.986648 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.986635    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj" event={"ID":"de172b2b-f10a-48dc-bed8-4b3ff0788ffb","Type":"ContainerDied","Data":"5c36f19a46cd912f15f2aeb351aa2193d2e935c61c473a398abe5f7fc43c333a"}
Oct 31 02:07:27.986897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:27.986721    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c36f19a46cd912f15f2aeb351aa2193d2e935c61c473a398abe5f7fc43c333a"
Oct 31 02:07:34.796173 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:34.796111    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"]
Oct 31 02:07:34.798270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:34.798239    5293 cpu_manager.go:414] "containerMap: removing container" podUID="de172b2b-f10a-48dc-bed8-4b3ff0788ffb" containerName="revoke-keys"
Oct 31 02:07:34.804112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:34.804083    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de172b2b-f10a-48dc-bed8-4b3ff0788ffb" containerName="revoke-keys"
Oct 31 02:07:34.804555 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:34.804536    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="de172b2b-f10a-48dc-bed8-4b3ff0788ffb" containerName="revoke-keys"
Oct 31 02:07:34.830030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:34.830000    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"]
Oct 31 02:07:34.830408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:34.830385    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:34.843380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:34.843346    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:34.843562 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:34.843365    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:07:35.227118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.205340    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghcs\" (UniqueName: \"kubernetes.io/projected/0515c49f-34cd-4eff-b7f0-d42e4070d93a-kube-api-access-2ghcs\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.227118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.205598    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.227118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.205756    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.356062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.355757    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.377118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.372292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghcs\" (UniqueName: \"kubernetes.io/projected/0515c49f-34cd-4eff-b7f0-d42e4070d93a-kube-api-access-2ghcs\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.377118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.372585    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.377118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.374298    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.405713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.405679    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.409338 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.409276    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghcs\" (UniqueName: \"kubernetes.io/projected/0515c49f-34cd-4eff-b7f0-d42e4070d93a-kube-api-access-2ghcs\") pod \"maas-api-revoke-sub-e9224db9-7xzhp\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.528688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.528607    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:35.836751 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:07:35.836683    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0515c49f_34cd_4eff_b7f0_d42e4070d93a.slice/crio-4af122584562c2a72acdc309ccba59e512d0b533c2be441b39740784184e2e93 WatchSource:0}: Error finding container 4af122584562c2a72acdc309ccba59e512d0b533c2be441b39740784184e2e93: Status 404 returned error can't find the container with id 4af122584562c2a72acdc309ccba59e512d0b533c2be441b39740784184e2e93
Oct 31 02:07:35.837239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.837210    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"]
Oct 31 02:07:35.970571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:35.970537    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:07:36.205087 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:36.205004    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp" event={"ID":"0515c49f-34cd-4eff-b7f0-d42e4070d93a","Type":"ContainerStarted","Data":"4af122584562c2a72acdc309ccba59e512d0b533c2be441b39740784184e2e93"}
Oct 31 02:07:37.216723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:37.216629    5293 generic.go:356] "Generic (PLEG): container finished" podID="0515c49f-34cd-4eff-b7f0-d42e4070d93a" containerID="0edc729b303871b338800bedaef2a8beff3af3ce7a9cbeb3ab4c3e736ee24c3c" exitCode=0
Oct 31 02:07:37.217141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:37.216717    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp" event={"ID":"0515c49f-34cd-4eff-b7f0-d42e4070d93a","Type":"ContainerDied","Data":"0edc729b303871b338800bedaef2a8beff3af3ce7a9cbeb3ab4c3e736ee24c3c"}
Oct 31 02:07:38.423155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.423123    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:38.609367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.609280    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0515c49f-34cd-4eff-b7f0-d42e4070d93a-kube-api-access-2ghcs\" (UniqueName: \"kubernetes.io/projected/0515c49f-34cd-4eff-b7f0-d42e4070d93a-kube-api-access-2ghcs\") pod \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") "
Oct 31 02:07:38.609551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.609375    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-xks-ca\") pod \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") "
Oct 31 02:07:38.609551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.609535    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-openshift-ca\") pod \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\" (UID: \"0515c49f-34cd-4eff-b7f0-d42e4070d93a\") "
Oct 31 02:07:38.609972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.609909    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-openshift-ca" pod "0515c49f-34cd-4eff-b7f0-d42e4070d93a" (UID: "0515c49f-34cd-4eff-b7f0-d42e4070d93a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:38.610646 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.610623    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:38.611826 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.611799    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-xks-ca" pod "0515c49f-34cd-4eff-b7f0-d42e4070d93a" (UID: "0515c49f-34cd-4eff-b7f0-d42e4070d93a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:38.612034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.612014    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0515c49f-34cd-4eff-b7f0-d42e4070d93a-kube-api-access-2ghcs" pod "0515c49f-34cd-4eff-b7f0-d42e4070d93a" (UID: "0515c49f-34cd-4eff-b7f0-d42e4070d93a"). InnerVolumeSpecName "kube-api-access-2ghcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:38.713434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.713395    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghcs\" (UniqueName: \"kubernetes.io/projected/0515c49f-34cd-4eff-b7f0-d42e4070d93a-kube-api-access-2ghcs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:38.713434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:38.713435    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0515c49f-34cd-4eff-b7f0-d42e4070d93a-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:39.319091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:39.316533    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp" event={"ID":"0515c49f-34cd-4eff-b7f0-d42e4070d93a","Type":"ContainerDied","Data":"4af122584562c2a72acdc309ccba59e512d0b533c2be441b39740784184e2e93"}
Oct 31 02:07:39.319091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:39.316597    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af122584562c2a72acdc309ccba59e512d0b533c2be441b39740784184e2e93"
Oct 31 02:07:39.319091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:39.316665    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"
Oct 31 02:07:41.828761 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:07:41.828720    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:07:42.669439 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.669409    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"]
Oct 31 02:07:42.670399 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.670375    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0515c49f-34cd-4eff-b7f0-d42e4070d93a" containerName="revoke-keys"
Oct 31 02:07:42.670399 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.670399    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0515c49f-34cd-4eff-b7f0-d42e4070d93a" containerName="revoke-keys"
Oct 31 02:07:42.670595 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.670570    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0515c49f-34cd-4eff-b7f0-d42e4070d93a" containerName="revoke-keys"
Oct 31 02:07:42.674436 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.674418    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.680594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.680571    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:42.680782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.680693    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:07:42.683538 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.683519    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"]
Oct 31 02:07:42.792052 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.792005    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.792242 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.792126    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.792382 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.792366    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbm2\" (UniqueName: \"kubernetes.io/projected/7b8c2477-c524-4630-9ac9-0b5edacad877-kube-api-access-lqbm2\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.894303 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.894268    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.894758 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.894347    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.894758 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.894654    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbm2\" (UniqueName: \"kubernetes.io/projected/7b8c2477-c524-4630-9ac9-0b5edacad877-kube-api-access-lqbm2\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.895193 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.895169    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.896949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.896927    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.906625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.906602    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbm2\" (UniqueName: \"kubernetes.io/projected/7b8c2477-c524-4630-9ac9-0b5edacad877-kube-api-access-lqbm2\") pod \"maas-api-revoke-sub-7100c22f-pklht\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:42.993519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:42.993445    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:43.325942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:43.325907    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"]
Oct 31 02:07:43.326860 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:07:43.326803    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8c2477_c524_4630_9ac9_0b5edacad877.slice/crio-4d97c5689f569866f733eb0d8957a649078e13ccd829418b564e9b2e6e3051bf WatchSource:0}: Error finding container 4d97c5689f569866f733eb0d8957a649078e13ccd829418b564e9b2e6e3051bf: Status 404 returned error can't find the container with id 4d97c5689f569866f733eb0d8957a649078e13ccd829418b564e9b2e6e3051bf
Oct 31 02:07:43.370720 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:43.370693    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht" event={"ID":"7b8c2477-c524-4630-9ac9-0b5edacad877","Type":"ContainerStarted","Data":"4d97c5689f569866f733eb0d8957a649078e13ccd829418b564e9b2e6e3051bf"}
Oct 31 02:07:44.382029 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:44.381995    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht" event={"ID":"7b8c2477-c524-4630-9ac9-0b5edacad877","Type":"ContainerStarted","Data":"ec589c41c21fcb936f4da802d5535f621c117bb550c14e3e68a8526b59bb7e38"}
Oct 31 02:07:44.399326 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:44.399273    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht" podStartSLOduration=2.399254942 podStartE2EDuration="2.399254942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:44.397759116 +0000 UTC m=+2822.346407508" watchObservedRunningTime="2026-10-31 02:07:44.399254942 +0000 UTC m=+2822.347903334"
Oct 31 02:07:45.422596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:45.422568    5293 generic.go:356] "Generic (PLEG): container finished" podID="7b8c2477-c524-4630-9ac9-0b5edacad877" containerID="ec589c41c21fcb936f4da802d5535f621c117bb550c14e3e68a8526b59bb7e38" exitCode=0
Oct 31 02:07:45.423230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:45.423184    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht" event={"ID":"7b8c2477-c524-4630-9ac9-0b5edacad877","Type":"ContainerDied","Data":"ec589c41c21fcb936f4da802d5535f621c117bb550c14e3e68a8526b59bb7e38"}
Oct 31 02:07:45.968825 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:45.968790    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:07:46.673426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.673391    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:46.784508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.784410    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-xks-ca\") pod \"7b8c2477-c524-4630-9ac9-0b5edacad877\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") "
Oct 31 02:07:46.784693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.784570    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8c2477-c524-4630-9ac9-0b5edacad877-kube-api-access-lqbm2\" (UniqueName: \"kubernetes.io/projected/7b8c2477-c524-4630-9ac9-0b5edacad877-kube-api-access-lqbm2\") pod \"7b8c2477-c524-4630-9ac9-0b5edacad877\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") "
Oct 31 02:07:46.784812 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.784790    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-openshift-ca\") pod \"7b8c2477-c524-4630-9ac9-0b5edacad877\" (UID: \"7b8c2477-c524-4630-9ac9-0b5edacad877\") "
Oct 31 02:07:46.785644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.785529    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-openshift-ca" pod "7b8c2477-c524-4630-9ac9-0b5edacad877" (UID: "7b8c2477-c524-4630-9ac9-0b5edacad877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:46.785885 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.785856    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:46.788233 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.788207    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8c2477-c524-4630-9ac9-0b5edacad877-kube-api-access-lqbm2" pod "7b8c2477-c524-4630-9ac9-0b5edacad877" (UID: "7b8c2477-c524-4630-9ac9-0b5edacad877"). InnerVolumeSpecName "kube-api-access-lqbm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:46.788825 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.788802    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-xks-ca" pod "7b8c2477-c524-4630-9ac9-0b5edacad877" (UID: "7b8c2477-c524-4630-9ac9-0b5edacad877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:46.795791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.795763    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:07:46.887399 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.887368    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8c2477-c524-4630-9ac9-0b5edacad877-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:46.887399 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:46.887395    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqbm2\" (UniqueName: \"kubernetes.io/projected/7b8c2477-c524-4630-9ac9-0b5edacad877-kube-api-access-lqbm2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:47.624075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:47.623564    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht" event={"ID":"7b8c2477-c524-4630-9ac9-0b5edacad877","Type":"ContainerDied","Data":"4d97c5689f569866f733eb0d8957a649078e13ccd829418b564e9b2e6e3051bf"}
Oct 31 02:07:47.624075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:47.623630    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d97c5689f569866f733eb0d8957a649078e13ccd829418b564e9b2e6e3051bf"
Oct 31 02:07:47.624075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:47.623701    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"
Oct 31 02:07:48.391703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:48.391666    5293 scope.go:122] "RemoveContainer" containerID="a20de8efad4ad0359d2b906c214e855814b11ad1a7397a106c7c2923521e0722"
Oct 31 02:07:48.575405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:48.572570    5293 scope.go:122] "RemoveContainer" containerID="4a33ee37d618cbd7f0c5be4b8dcae5ab563da7df64a21acc009ff421d3db3411"
Oct 31 02:07:49.483278 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:49.483233    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:07:49.866116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:49.864661    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:07:54.507573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.507531    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"]
Oct 31 02:07:54.509408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.509380    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7b8c2477-c524-4630-9ac9-0b5edacad877" containerName="revoke-keys"
Oct 31 02:07:54.509558 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.509545    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8c2477-c524-4630-9ac9-0b5edacad877" containerName="revoke-keys"
Oct 31 02:07:54.509915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.509897    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8c2477-c524-4630-9ac9-0b5edacad877" containerName="revoke-keys"
Oct 31 02:07:54.550960 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.550932    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.557986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.557934    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:07:54.570105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.570073    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"]
Oct 31 02:07:54.626921 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.586304    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:54.720785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.720745    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.750066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.733276    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.771471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.771377    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwtp\" (UniqueName: \"kubernetes.io/projected/95c13248-c64a-424b-8312-f4ac5e345345-kube-api-access-scwtp\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.931832 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.931791    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwtp\" (UniqueName: \"kubernetes.io/projected/95c13248-c64a-424b-8312-f4ac5e345345-kube-api-access-scwtp\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.932029 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.931944    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.932243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.932223    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.933086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.933023    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.935327 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.935306    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:54.943720 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:54.943669    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwtp\" (UniqueName: \"kubernetes.io/projected/95c13248-c64a-424b-8312-f4ac5e345345-kube-api-access-scwtp\") pod \"maas-api-revoke-sub-7031b904-kwnpq\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:55.075497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:55.071814    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:55.970884 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:55.970843    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:07:56.038409 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:56.038318    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"]
Oct 31 02:07:57.037142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:57.031402    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq" event={"ID":"95c13248-c64a-424b-8312-f4ac5e345345","Type":"ContainerStarted","Data":"60c7b614bc63b229a7292fee9f7ec979ae8c26be6769a5d703a594dd7b5ec5d2"}
Oct 31 02:07:57.037142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:57.031474    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq" event={"ID":"95c13248-c64a-424b-8312-f4ac5e345345","Type":"ContainerStarted","Data":"e53c69fb9c61163315080050c01b189ca9720090a1041a805408dbb79c08a275"}
Oct 31 02:07:57.096150 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:57.095257    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq" podStartSLOduration=3.095231764 podStartE2EDuration="3.095231764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:57.058194923 +0000 UTC m=+2835.006843328" watchObservedRunningTime="2026-10-31 02:07:57.095231764 +0000 UTC m=+2835.043880156"
Oct 31 02:07:58.042110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:58.042068    5293 generic.go:356] "Generic (PLEG): container finished" podID="95c13248-c64a-424b-8312-f4ac5e345345" containerID="60c7b614bc63b229a7292fee9f7ec979ae8c26be6769a5d703a594dd7b5ec5d2" exitCode=0
Oct 31 02:07:58.042604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:58.042135    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq" event={"ID":"95c13248-c64a-424b-8312-f4ac5e345345","Type":"ContainerDied","Data":"60c7b614bc63b229a7292fee9f7ec979ae8c26be6769a5d703a594dd7b5ec5d2"}
Oct 31 02:07:59.254541 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.254512    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:07:59.371834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.371758    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-openshift-ca\") pod \"95c13248-c64a-424b-8312-f4ac5e345345\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") "
Oct 31 02:07:59.371834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.371809    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-xks-ca\") pod \"95c13248-c64a-424b-8312-f4ac5e345345\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") "
Oct 31 02:07:59.372085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.371841    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c13248-c64a-424b-8312-f4ac5e345345-kube-api-access-scwtp\" (UniqueName: \"kubernetes.io/projected/95c13248-c64a-424b-8312-f4ac5e345345-kube-api-access-scwtp\") pod \"95c13248-c64a-424b-8312-f4ac5e345345\" (UID: \"95c13248-c64a-424b-8312-f4ac5e345345\") "
Oct 31 02:07:59.372162 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.372138    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-openshift-ca" pod "95c13248-c64a-424b-8312-f4ac5e345345" (UID: "95c13248-c64a-424b-8312-f4ac5e345345"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:59.372685 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.372654    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:59.374164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.374143    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-xks-ca" pod "95c13248-c64a-424b-8312-f4ac5e345345" (UID: "95c13248-c64a-424b-8312-f4ac5e345345"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:59.374278 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.374234    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c13248-c64a-424b-8312-f4ac5e345345-kube-api-access-scwtp" pod "95c13248-c64a-424b-8312-f4ac5e345345" (UID: "95c13248-c64a-424b-8312-f4ac5e345345"). InnerVolumeSpecName "kube-api-access-scwtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:59.474322 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.474288    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c13248-c64a-424b-8312-f4ac5e345345-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:59.474322 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:07:59.474317    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scwtp\" (UniqueName: \"kubernetes.io/projected/95c13248-c64a-424b-8312-f4ac5e345345-kube-api-access-scwtp\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:00.061221 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:00.061187    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq" event={"ID":"95c13248-c64a-424b-8312-f4ac5e345345","Type":"ContainerDied","Data":"e53c69fb9c61163315080050c01b189ca9720090a1041a805408dbb79c08a275"}
Oct 31 02:08:00.061508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:00.061235    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e53c69fb9c61163315080050c01b189ca9720090a1041a805408dbb79c08a275"
Oct 31 02:08:00.061508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:00.061202    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"
Oct 31 02:08:04.702112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:04.702064    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:04.986120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:04.937666    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:04.989169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:04.988020    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:05.030521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:05.030456    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"]
Oct 31 02:08:05.031115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:05.030809    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:05.061600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:05.061165    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:08:05.228277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:05.228229    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:05.735240 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:05.735203    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"]
Oct 31 02:08:07.009062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:07.008103    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:08:07.009616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:07.009142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:07.101070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:07.100781    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 02:08:07.122470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:07.122425    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:07.143063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:07.141581    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:07.220235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:07.212708    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:08:07.278584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:07.278488    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:08:07.479477 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:07.471955    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:08.071068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.065802    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:08.071068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.066662    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:08:08.144067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.124017    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:08.144067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.124614    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:08:08.163067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.153264    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:08:08.166881 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.166849    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:08:08.225112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.225074    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:08.303940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.303867    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:08.788067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:08.782316    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:08:09.050189 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.025092    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:08:09.064605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.064513    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:08:09.075875 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.071332    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:09.120424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.118745    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:08:09.120716 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.120700    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:08:09.171385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.169733    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:08:09.171385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.169783    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:08:09.171385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.169854    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:08:09.171385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.169890    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:08:09.171385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.127567    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:09.173796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.167836    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:09.187061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.180220    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:08:09.227104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.227027    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:08:09.228178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:09.228153    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:08:10.031486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.031438    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerName="payload-processing" containerID="cri-o://8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96" gracePeriod=25
Oct 31 02:08:10.524470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.524444    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:08:10.630952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.630864    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-provider-credentials\") pod \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") "
Oct 31 02:08:10.630952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.630905    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-kube-api-access-ss9k8\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-kube-api-access-ss9k8\") pod \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") "
Oct 31 02:08:10.631248 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.631097    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-plugins-config-volume\") pod \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") "
Oct 31 02:08:10.631339 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.631325    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-routing-overlay-volume\") pod \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\" (UID: \"93a9a213-af7f-47c7-b31d-d021c33f2d2b\") "
Oct 31 02:08:10.631541 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.631516    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-plugins-config-volume" pod "93a9a213-af7f-47c7-b31d-d021c33f2d2b" (UID: "93a9a213-af7f-47c7-b31d-d021c33f2d2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:10.631888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.631860    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-routing-overlay-volume" pod "93a9a213-af7f-47c7-b31d-d021c33f2d2b" (UID: "93a9a213-af7f-47c7-b31d-d021c33f2d2b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:10.632551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.632528    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:10.632624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.632559    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a9a213-af7f-47c7-b31d-d021c33f2d2b-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:10.633509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.633484    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-kube-api-access-ss9k8" pod "93a9a213-af7f-47c7-b31d-d021c33f2d2b" (UID: "93a9a213-af7f-47c7-b31d-d021c33f2d2b"). InnerVolumeSpecName "kube-api-access-ss9k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:10.633611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.633522    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-provider-credentials" pod "93a9a213-af7f-47c7-b31d-d021c33f2d2b" (UID: "93a9a213-af7f-47c7-b31d-d021c33f2d2b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:10.734023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.733987    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:10.734023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.734016    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9k8\" (UniqueName: \"kubernetes.io/projected/93a9a213-af7f-47c7-b31d-d021c33f2d2b-kube-api-access-ss9k8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:10.736987 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.736952    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5" podUID="10e20f0e-1fef-45d0-9585-409d7ef385c6" containerName="payload-processing" containerID="cri-o://2e53b7434b82a717f9ff727a4d364073c8335ab96a418603ea02e0c43dc2fcd1" gracePeriod=25
Oct 31 02:08:10.963621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.963584    5293 generic.go:356] "Generic (PLEG): container finished" podID="10e20f0e-1fef-45d0-9585-409d7ef385c6" containerID="2e53b7434b82a717f9ff727a4d364073c8335ab96a418603ea02e0c43dc2fcd1" exitCode=0
Oct 31 02:08:10.964008 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.963976    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5" event={"ID":"10e20f0e-1fef-45d0-9585-409d7ef385c6","Type":"ContainerDied","Data":"2e53b7434b82a717f9ff727a4d364073c8335ab96a418603ea02e0c43dc2fcd1"}
Oct 31 02:08:10.984548 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.984512    5293 generic.go:356] "Generic (PLEG): container finished" podID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerID="8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96" exitCode=0
Oct 31 02:08:10.985295 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.984830    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"
Oct 31 02:08:10.986194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.984851    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" event={"ID":"93a9a213-af7f-47c7-b31d-d021c33f2d2b","Type":"ContainerDied","Data":"8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96"}
Oct 31 02:08:10.993061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.986385    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9" event={"ID":"93a9a213-af7f-47c7-b31d-d021c33f2d2b","Type":"ContainerDied","Data":"6a099926f6a4c4b272f2304ebe011e999f27a2cefc207c4196094ab4f21ffb60"}
Oct 31 02:08:10.994186 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:10.988149    5293 scope.go:122] "RemoveContainer" containerID="8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96"
Oct 31 02:08:11.024684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.024340    5293 scope.go:122] "RemoveContainer" containerID="8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96"
Oct 31 02:08:11.025128 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:08:11.024948    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96\": container with ID starting with 8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96 not found: ID does not exist" containerID="8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96"
Oct 31 02:08:11.025128 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.024987    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96"} err="failed to get container status \"8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96\": rpc error: code = NotFound desc = could not find container \"8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96\": container with ID starting with 8c32d2bb6fcb65a3e0983539f397c3e52112d1903987812a22ae1f76dc826c96 not found: ID does not exist"
Oct 31 02:08:11.050130 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.050095    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"]
Oct 31 02:08:11.057795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.057775    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-84bbtdkt9"]
Oct 31 02:08:11.209880 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.209854    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:08:11.246617 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.246583    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-provider-credentials\") pod \"10e20f0e-1fef-45d0-9585-409d7ef385c6\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") "
Oct 31 02:08:11.246817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.246636    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"]
Oct 31 02:08:11.246817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.246780    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-kube-api-access-h8wqg\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-kube-api-access-h8wqg\") pod \"10e20f0e-1fef-45d0-9585-409d7ef385c6\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") "
Oct 31 02:08:11.246954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.246868    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-plugins-config-volume\") pod \"10e20f0e-1fef-45d0-9585-409d7ef385c6\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") "
Oct 31 02:08:11.247270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.247243    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-routing-overlay-volume\") pod \"10e20f0e-1fef-45d0-9585-409d7ef385c6\" (UID: \"10e20f0e-1fef-45d0-9585-409d7ef385c6\") "
Oct 31 02:08:11.247790 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.247759    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-routing-overlay-volume" pod "10e20f0e-1fef-45d0-9585-409d7ef385c6" (UID: "10e20f0e-1fef-45d0-9585-409d7ef385c6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:11.248141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248119    5293 cpu_manager.go:414] "containerMap: removing container" podUID="10e20f0e-1fef-45d0-9585-409d7ef385c6" containerName="payload-processing"
Oct 31 02:08:11.248250 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248144    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e20f0e-1fef-45d0-9585-409d7ef385c6" containerName="payload-processing"
Oct 31 02:08:11.248250 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248213    5293 cpu_manager.go:414] "containerMap: removing container" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerName="payload-processing"
Oct 31 02:08:11.248250 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248222    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerName="payload-processing"
Oct 31 02:08:11.248250 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248250    5293 cpu_manager.go:414] "containerMap: removing container" podUID="95c13248-c64a-424b-8312-f4ac5e345345" containerName="revoke-keys"
Oct 31 02:08:11.248478 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248258    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c13248-c64a-424b-8312-f4ac5e345345" containerName="revoke-keys"
Oct 31 02:08:11.248478 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248469    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c13248-c64a-424b-8312-f4ac5e345345" containerName="revoke-keys"
Oct 31 02:08:11.248583 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248490    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e20f0e-1fef-45d0-9585-409d7ef385c6" containerName="payload-processing"
Oct 31 02:08:11.248583 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.248508    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" containerName="payload-processing"
Oct 31 02:08:11.249242 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.249220    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:11.250213 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.250187    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-plugins-config-volume" pod "10e20f0e-1fef-45d0-9585-409d7ef385c6" (UID: "10e20f0e-1fef-45d0-9585-409d7ef385c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:11.251154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.250996    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-provider-credentials" pod "10e20f0e-1fef-45d0-9585-409d7ef385c6" (UID: "10e20f0e-1fef-45d0-9585-409d7ef385c6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:11.251154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.251118    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-kube-api-access-h8wqg" pod "10e20f0e-1fef-45d0-9585-409d7ef385c6" (UID: "10e20f0e-1fef-45d0-9585-409d7ef385c6"). InnerVolumeSpecName "kube-api-access-h8wqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:11.252862 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.252846    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.268350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.265242    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"]
Oct 31 02:08:11.351291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351258    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-token\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351469 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351321    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7xq\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-kube-api-access-5h7xq\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351469 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351422    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-envoy\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351473    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-credential-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351544    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167f5c4-6e60-4517-8904-450dd8e8416c-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351614    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-certs\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351770 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351727    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-podinfo\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351767    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-data\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351804    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.351957 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351945    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wqg\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-kube-api-access-h8wqg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:11.352011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351960    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e20f0e-1fef-45d0-9585-409d7ef385c6-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:11.352011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.351970    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e20f0e-1fef-45d0-9585-409d7ef385c6-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:11.453426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453335    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-data\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.453426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453396    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.453681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453558    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-token\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.453681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453603    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7xq\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-kube-api-access-5h7xq\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.453681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453640    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-envoy\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.453681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453673    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-credential-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.454243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453765    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167f5c4-6e60-4517-8904-450dd8e8416c-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.454243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453796    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.454243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453868    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-data\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.454243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.453881    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-certs\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.454243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.454114    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-podinfo\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.454243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.454127    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-credential-socket\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.454243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.454168    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-certs\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.454554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.454444    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167f5c4-6e60-4517-8904-450dd8e8416c-istiod-ca-cert\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.456723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.456699    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-envoy\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.457215 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.457193    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-podinfo\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.465381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.465358    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-token\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.466486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.466462    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7xq\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-kube-api-access-5h7xq\") pod \"e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") " pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.616720 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.616375    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:11.794260 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:08:11.794196    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:08:11.986807 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:08:11.986776    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd167f5c4_6e60_4517_8904_450dd8e8416c.slice/crio-a44d95799b29669596ce4e4ac609e8dfef0cb84623afef918246db89c0156d84 WatchSource:0}: Error finding container a44d95799b29669596ce4e4ac609e8dfef0cb84623afef918246db89c0156d84: Status 404 returned error can't find the container with id a44d95799b29669596ce4e4ac609e8dfef0cb84623afef918246db89c0156d84
Oct 31 02:08:11.987207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.987189    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"]
Oct 31 02:08:11.989915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.989878    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:08:11.990033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.989957    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:08:11.990033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.990011    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:08:11.997091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.997014    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5" event={"ID":"10e20f0e-1fef-45d0-9585-409d7ef385c6","Type":"ContainerDied","Data":"cc839a8d1c1225d4f4ac2d97816c8786574ad8cb79f989bd78d857cceaa932c1"}
Oct 31 02:08:11.997188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.997125    5293 scope.go:122] "RemoveContainer" containerID="2e53b7434b82a717f9ff727a4d364073c8335ab96a418603ea02e0c43dc2fcd1"
Oct 31 02:08:11.997261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:11.997232    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"
Oct 31 02:08:12.025149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:12.025120    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"]
Oct 31 02:08:12.030338 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:12.030310    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5796scds5"]
Oct 31 02:08:12.178432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:12.178400    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e20f0e-1fef-45d0-9585-409d7ef385c6" path="/var/lib/kubelet/pods/10e20f0e-1fef-45d0-9585-409d7ef385c6/volumes"
Oct 31 02:08:12.178863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:12.178836    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a9a213-af7f-47c7-b31d-d021c33f2d2b" path="/var/lib/kubelet/pods/93a9a213-af7f-47c7-b31d-d021c33f2d2b/volumes"
Oct 31 02:08:13.054951 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:13.054911    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" event={"ID":"d167f5c4-6e60-4517-8904-450dd8e8416c","Type":"ContainerStarted","Data":"462f9ce23423650f45ed18d38fcdf3eb134d24c0d3d6df35bbf38c04c9ea8c66"}
Oct 31 02:08:13.055390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:13.054983    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" event={"ID":"d167f5c4-6e60-4517-8904-450dd8e8416c","Type":"ContainerStarted","Data":"a44d95799b29669596ce4e4ac609e8dfef0cb84623afef918246db89c0156d84"}
Oct 31 02:08:13.083541 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:13.083491    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podStartSLOduration=151.083475556 podStartE2EDuration="2m31.083475556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:13.079489002 +0000 UTC m=+2851.028137395" watchObservedRunningTime="2026-10-31 02:08:13.083475556 +0000 UTC m=+2851.032123947"
Oct 31 02:08:13.627064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:13.619957    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Oct 31 02:08:13.627064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:13.620056    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Oct 31 02:08:13.627064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:13.620364    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:14.686808 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:14.686765    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Oct 31 02:08:14.687597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:14.687559    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Oct 31 02:08:15.627770 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:15.627402    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Oct 31 02:08:15.627770 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:15.627469    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Oct 31 02:08:16.637214 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:16.637175    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Oct 31 02:08:16.647174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:16.647130    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Oct 31 02:08:17.628104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:17.625442    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Oct 31 02:08:17.628104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:17.625522    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Oct 31 02:08:18.637492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:18.637450    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Oct 31 02:08:18.638010 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:18.637521    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Oct 31 02:08:19.621062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:19.618927    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Oct 31 02:08:19.621062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:19.619000    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Oct 31 02:08:19.872118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:19.872000    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:20.039828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.037309    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:08:20.182104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.176152    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 02:08:20.196402 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.196351    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:08:20.213071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.206506    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:20.216110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.215302    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:08:20.255225 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.255166    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:20.275500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.271440    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:20.275500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.271850    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:20.275500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.272123    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:08:20.275500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.274168    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:20.275500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.274436    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 02:08:20.308050 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.307981    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 02:08:20.656211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.656173    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Oct 31 02:08:20.656494 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:20.656468    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Oct 31 02:08:21.687703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:21.687673    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:21.688448 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:21.688426    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:21.689393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:21.689373    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:08:23.988071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:23.986134    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:23.988588 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:23.988194    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:30.155071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:30.149436    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:30.155071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:30.149489    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 02:08:30.155071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:30.149759    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:30.199776 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:30.199243    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:30.237089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:30.230234    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:30.237089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:30.230493    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:30.237089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:30.236174    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:41.781099 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:08:41.781059    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:08:52.505370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:08:52.505162    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:09:11.781353 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:09:11.781315    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:09:28.536777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.536744    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"]
Oct 31 02:09:28.542185 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.542166    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.547316 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.547292    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:09:28.547493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.547404    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:09:28.557333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.557307    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"]
Oct 31 02:09:28.646183 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.646154    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.646389 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.646319    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prr4p\" (UniqueName: \"kubernetes.io/projected/819ff781-73a2-4758-af55-e8b8fbcdba3d-kube-api-access-prr4p\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.646491 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.646475    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.747752 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.747716    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prr4p\" (UniqueName: \"kubernetes.io/projected/819ff781-73a2-4758-af55-e8b8fbcdba3d-kube-api-access-prr4p\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.747940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.747916    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.748089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.748069    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.748586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.748562    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.750636 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.750618    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.760925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.760902    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prr4p\" (UniqueName: \"kubernetes.io/projected/819ff781-73a2-4758-af55-e8b8fbcdba3d-kube-api-access-prr4p\") pod \"maas-api-revoke-sub-b9308413-kk779\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:28.873123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:28.872993    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:29.205997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:29.201579    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"]
Oct 31 02:09:29.265551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:29.265520    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779" event={"ID":"819ff781-73a2-4758-af55-e8b8fbcdba3d","Type":"ContainerStarted","Data":"de765cdea745ba3f66ff7b6942a0ac78daf807e79a9d62bffe6e54d71353a4f0"}
Oct 31 02:09:30.276443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:30.276413    5293 generic.go:356] "Generic (PLEG): container finished" podID="819ff781-73a2-4758-af55-e8b8fbcdba3d" containerID="c7142f62f24a30b9ed51ffba0a796d5372080c46bb6c61c0056814b21a545a13" exitCode=0
Oct 31 02:09:30.276833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:30.276493    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779" event={"ID":"819ff781-73a2-4758-af55-e8b8fbcdba3d","Type":"ContainerDied","Data":"c7142f62f24a30b9ed51ffba0a796d5372080c46bb6c61c0056814b21a545a13"}
Oct 31 02:09:31.494277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.494247    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:31.547836 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.547753    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-xks-ca\") pod \"819ff781-73a2-4758-af55-e8b8fbcdba3d\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") "
Oct 31 02:09:31.547836 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.547800    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819ff781-73a2-4758-af55-e8b8fbcdba3d-kube-api-access-prr4p\" (UniqueName: \"kubernetes.io/projected/819ff781-73a2-4758-af55-e8b8fbcdba3d-kube-api-access-prr4p\") pod \"819ff781-73a2-4758-af55-e8b8fbcdba3d\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") "
Oct 31 02:09:31.548127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.547842    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-openshift-ca\") pod \"819ff781-73a2-4758-af55-e8b8fbcdba3d\" (UID: \"819ff781-73a2-4758-af55-e8b8fbcdba3d\") "
Oct 31 02:09:31.548519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.548483    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-openshift-ca" pod "819ff781-73a2-4758-af55-e8b8fbcdba3d" (UID: "819ff781-73a2-4758-af55-e8b8fbcdba3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:31.549073 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.549014    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:09:31.550344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.550316    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-xks-ca" pod "819ff781-73a2-4758-af55-e8b8fbcdba3d" (UID: "819ff781-73a2-4758-af55-e8b8fbcdba3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:31.550508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.550434    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819ff781-73a2-4758-af55-e8b8fbcdba3d-kube-api-access-prr4p" pod "819ff781-73a2-4758-af55-e8b8fbcdba3d" (UID: "819ff781-73a2-4758-af55-e8b8fbcdba3d"). InnerVolumeSpecName "kube-api-access-prr4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:31.651384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.651349    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819ff781-73a2-4758-af55-e8b8fbcdba3d-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:09:31.651384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:31.651379    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prr4p\" (UniqueName: \"kubernetes.io/projected/819ff781-73a2-4758-af55-e8b8fbcdba3d-kube-api-access-prr4p\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.294563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:32.294534    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"
Oct 31 02:09:32.294785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:32.294569    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779" event={"ID":"819ff781-73a2-4758-af55-e8b8fbcdba3d","Type":"ContainerDied","Data":"de765cdea745ba3f66ff7b6942a0ac78daf807e79a9d62bffe6e54d71353a4f0"}
Oct 31 02:09:32.294785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:32.294615    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de765cdea745ba3f66ff7b6942a0ac78daf807e79a9d62bffe6e54d71353a4f0"
Oct 31 02:09:37.027309 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:37.027275    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"]
Oct 31 02:09:37.032665 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:37.032632    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafdfd2f-5ms8f"]
Oct 31 02:09:38.205706 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:38.205664    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad39557d-531b-486d-90f8-c63f8b8e12fa" path="/var/lib/kubelet/pods/ad39557d-531b-486d-90f8-c63f8b8e12fa/volumes"
Oct 31 02:09:39.121472 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.121438    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"]
Oct 31 02:09:39.122812 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.122786    5293 cpu_manager.go:414] "containerMap: removing container" podUID="819ff781-73a2-4758-af55-e8b8fbcdba3d" containerName="revoke-keys"
Oct 31 02:09:39.122812 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.122811    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819ff781-73a2-4758-af55-e8b8fbcdba3d" containerName="revoke-keys"
Oct 31 02:09:39.123068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.123054    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="819ff781-73a2-4758-af55-e8b8fbcdba3d" containerName="revoke-keys"
Oct 31 02:09:39.130055 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.130017    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.133571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.133547    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:09:39.134233 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.134156    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:09:39.136512 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.136362    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"]
Oct 31 02:09:39.239449 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.239416    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwzzs\" (UniqueName: \"kubernetes.io/projected/2722d7a2-227c-4d59-b497-1d48c4aa896b-kube-api-access-wwzzs\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.239938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.239598    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.239938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.239879    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.342097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.342033    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.342341 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.342323    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwzzs\" (UniqueName: \"kubernetes.io/projected/2722d7a2-227c-4d59-b497-1d48c4aa896b-kube-api-access-wwzzs\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.342416 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.342403    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.343132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.343104    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.345248 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.345225    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.353762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.353738    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwzzs\" (UniqueName: \"kubernetes.io/projected/2722d7a2-227c-4d59-b497-1d48c4aa896b-kube-api-access-wwzzs\") pod \"maas-api-revoke-sub-47681e17-nhxf7\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.451694 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.451660    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:39.952251 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:09:39.952216    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2722d7a2_227c_4d59_b497_1d48c4aa896b.slice/crio-59edfe2e554df71ca62031bd8e43ed0c14360b7625d21edb096e97db8dd0704c WatchSource:0}: Error finding container 59edfe2e554df71ca62031bd8e43ed0c14360b7625d21edb096e97db8dd0704c: Status 404 returned error can't find the container with id 59edfe2e554df71ca62031bd8e43ed0c14360b7625d21edb096e97db8dd0704c
Oct 31 02:09:39.952997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:39.952966    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"]
Oct 31 02:09:40.483249 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:40.483213    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7" event={"ID":"2722d7a2-227c-4d59-b497-1d48c4aa896b","Type":"ContainerStarted","Data":"d43c853cb2b8e9a0a466dc8eac3e70d880d69c7e187f7bf1df75177031a0e799"}
Oct 31 02:09:40.483669 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:40.483275    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7" event={"ID":"2722d7a2-227c-4d59-b497-1d48c4aa896b","Type":"ContainerStarted","Data":"59edfe2e554df71ca62031bd8e43ed0c14360b7625d21edb096e97db8dd0704c"}
Oct 31 02:09:40.505329 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:40.505246    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7" podStartSLOduration=1.50523088 podStartE2EDuration="1.50523088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:09:40.499863568 +0000 UTC m=+2938.448511961" watchObservedRunningTime="2026-10-31 02:09:40.50523088 +0000 UTC m=+2938.453879271"
Oct 31 02:09:41.493186 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:41.493139    5293 generic.go:356] "Generic (PLEG): container finished" podID="2722d7a2-227c-4d59-b497-1d48c4aa896b" containerID="d43c853cb2b8e9a0a466dc8eac3e70d880d69c7e187f7bf1df75177031a0e799" exitCode=0
Oct 31 02:09:41.493613 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:41.493205    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7" event={"ID":"2722d7a2-227c-4d59-b497-1d48c4aa896b","Type":"ContainerDied","Data":"d43c853cb2b8e9a0a466dc8eac3e70d880d69c7e187f7bf1df75177031a0e799"}
Oct 31 02:09:41.782922 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:09:41.782835    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:09:42.689445 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.689419    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:42.771770 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.771684    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2722d7a2-227c-4d59-b497-1d48c4aa896b-kube-api-access-wwzzs\" (UniqueName: \"kubernetes.io/projected/2722d7a2-227c-4d59-b497-1d48c4aa896b-kube-api-access-wwzzs\") pod \"2722d7a2-227c-4d59-b497-1d48c4aa896b\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") "
Oct 31 02:09:42.771937 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.771846    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-openshift-ca\") pod \"2722d7a2-227c-4d59-b497-1d48c4aa896b\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") "
Oct 31 02:09:42.772117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.772097    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-xks-ca\") pod \"2722d7a2-227c-4d59-b497-1d48c4aa896b\" (UID: \"2722d7a2-227c-4d59-b497-1d48c4aa896b\") "
Oct 31 02:09:42.772353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.772322    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-openshift-ca" pod "2722d7a2-227c-4d59-b497-1d48c4aa896b" (UID: "2722d7a2-227c-4d59-b497-1d48c4aa896b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:42.773162 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.773006    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:09:42.774553 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.774528    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2722d7a2-227c-4d59-b497-1d48c4aa896b-kube-api-access-wwzzs" pod "2722d7a2-227c-4d59-b497-1d48c4aa896b" (UID: "2722d7a2-227c-4d59-b497-1d48c4aa896b"). InnerVolumeSpecName "kube-api-access-wwzzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:42.774788 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.774763    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-xks-ca" pod "2722d7a2-227c-4d59-b497-1d48c4aa896b" (UID: "2722d7a2-227c-4d59-b497-1d48c4aa896b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:42.875674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.875642    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2722d7a2-227c-4d59-b497-1d48c4aa896b-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:09:42.875674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:42.875674    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwzzs\" (UniqueName: \"kubernetes.io/projected/2722d7a2-227c-4d59-b497-1d48c4aa896b-kube-api-access-wwzzs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:09:43.517486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:43.517451    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7" event={"ID":"2722d7a2-227c-4d59-b497-1d48c4aa896b","Type":"ContainerDied","Data":"59edfe2e554df71ca62031bd8e43ed0c14360b7625d21edb096e97db8dd0704c"}
Oct 31 02:09:43.517486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:43.517487    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"
Oct 31 02:09:43.517761 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:43.517502    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59edfe2e554df71ca62031bd8e43ed0c14360b7625d21edb096e97db8dd0704c"
Oct 31 02:09:46.815196 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:46.815158    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:09:47.028146 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:47.028105    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"]
Oct 31 02:09:47.034181 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:47.034151    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66e0592-jmhmm"]
Oct 31 02:09:48.239064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:48.226069    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68f815a-98a1-4214-baba-30b97e893687" path="/var/lib/kubelet/pods/f68f815a-98a1-4214-baba-30b97e893687/volumes"
Oct 31 02:09:52.052065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:52.051113    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"]
Oct 31 02:09:52.094940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:52.094078    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1989431f-nxmrf"]
Oct 31 02:09:52.177348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:52.177314    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3cf7cf9-7810-4084-b25e-b2eb7768bf0b" path="/var/lib/kubelet/pods/d3cf7cf9-7810-4084-b25e-b2eb7768bf0b/volumes"
Oct 31 02:09:59.032161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:59.032121    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"]
Oct 31 02:09:59.037965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:09:59.037932    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fac276b8-ts9rc"]
Oct 31 02:10:00.179768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:00.179730    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c96b803-3046-4e4a-a5a5-76c1a2adde1a" path="/var/lib/kubelet/pods/8c96b803-3046-4e4a-a5a5-76c1a2adde1a/volumes"
Oct 31 02:10:07.111952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:07.111920    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"]
Oct 31 02:10:07.233182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:07.203789    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92535836-bfgs6"]
Oct 31 02:10:08.176688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:08.176653    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609e08ae-ddf0-43a0-8854-110ccdcb2f02" path="/var/lib/kubelet/pods/609e08ae-ddf0-43a0-8854-110ccdcb2f02/volumes"
Oct 31 02:10:11.782076 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:10:11.782019    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:10:31.055597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:31.055508    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"]
Oct 31 02:10:31.086359 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:31.086303    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b89af60-qtrpq"]
Oct 31 02:10:32.179560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:32.179512    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5c56d2-712a-48f2-97f0-7fdabd381adf" path="/var/lib/kubelet/pods/7a5c56d2-712a-48f2-97f0-7fdabd381adf/volumes"
Oct 31 02:10:34.455845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:34.455777    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"]
Oct 31 02:10:34.456692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:34.456655    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy" containerID="cri-o://ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b" gracePeriod=30
Oct 31 02:10:38.720536 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:38.720501    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"]
Oct 31 02:10:38.721163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:38.720710    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy" containerID="cri-o://db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088" gracePeriod=30
Oct 31 02:10:39.779208 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.779160    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:10:39.816643 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.816609    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"]
Oct 31 02:10:39.817662 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.817638    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2722d7a2-227c-4d59-b497-1d48c4aa896b" containerName="revoke-keys"
Oct 31 02:10:39.817662 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.817660    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2722d7a2-227c-4d59-b497-1d48c4aa896b" containerName="revoke-keys"
Oct 31 02:10:39.817824 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.817688    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy"
Oct 31 02:10:39.817824 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.817693    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy"
Oct 31 02:10:39.817911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.817847    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerName="istio-proxy"
Oct 31 02:10:39.817911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.817873    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2722d7a2-227c-4d59-b497-1d48c4aa896b" containerName="revoke-keys"
Oct 31 02:10:39.821896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.821877    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:39.833385 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.833354    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-data\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.833629 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.833608    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-kube-api-access-gnwfw\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-kube-api-access-gnwfw\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.833914 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.833892    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istiod-ca-cert\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.834210 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.834189    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-credential-socket\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.834464 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.834447    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-envoy\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.834606 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.834589    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-certs\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.834870 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.834851    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-podinfo\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.835065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.835028    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-socket\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.835228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.835210    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-token\") pod \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\" (UID: \"0cc6ec04-1cc3-4859-a53b-10fc82ff3433\") "
Oct 31 02:10:39.842094 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.837232    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-credential-socket" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:39.842094 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.837600    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-data" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:39.842094 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.840529    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-kube-api-access-gnwfw" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "kube-api-access-gnwfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:39.842094 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.840836    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istiod-ca-cert" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:39.842094 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.841116    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-certs" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:39.843059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.842609    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-91b6b1-serving-cert\""
Oct 31 02:10:39.843059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.842729    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-socket" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:39.845714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.845687    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-envoy" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:39.846807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.846787    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-token" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:39.847406 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.847383    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"]
Oct 31 02:10:39.848550 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.848516    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-podinfo" pod "0cc6ec04-1cc3-4859-a53b-10fc82ff3433" (UID: "0cc6ec04-1cc3-4859-a53b-10fc82ff3433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:10:39.939245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939186    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:39.939404 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939372    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18654831-3ecb-46ae-bd66-59c4725008fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:39.939454 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939419    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvpm\" (UniqueName: \"kubernetes.io/projected/18654831-3ecb-46ae-bd66-59c4725008fe-kube-api-access-ztvpm\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:39.939649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939623    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:39.939796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939760    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.939796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939773    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.939796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939782    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.939796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939790    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.939796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939799    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.940083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939808    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.940083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939816    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.940083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939824    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.940083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:39.939832    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnwfw\" (UniqueName: \"kubernetes.io/projected/0cc6ec04-1cc3-4859-a53b-10fc82ff3433-kube-api-access-gnwfw\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.054104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.053977    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.054800 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.054769    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.055321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.055296    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18654831-3ecb-46ae-bd66-59c4725008fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.055557 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.055519    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvpm\" (UniqueName: \"kubernetes.io/projected/18654831-3ecb-46ae-bd66-59c4725008fe-kube-api-access-ztvpm\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.057248 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.057229    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18654831-3ecb-46ae-bd66-59c4725008fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.094224 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.094158    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.094976 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.094925    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.095704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.095683    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvpm\" (UniqueName: \"kubernetes.io/projected/18654831-3ecb-46ae-bd66-59c4725008fe-kube-api-access-ztvpm\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.098844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.097431    5293 generic.go:356] "Generic (PLEG): container finished" podID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" containerID="ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b" exitCode=0
Oct 31 02:10:40.098844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.097551    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"
Oct 31 02:10:40.098844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.097548    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" event={"ID":"0cc6ec04-1cc3-4859-a53b-10fc82ff3433","Type":"ContainerDied","Data":"ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b"}
Oct 31 02:10:40.098844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.097694    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk" event={"ID":"0cc6ec04-1cc3-4859-a53b-10fc82ff3433","Type":"ContainerDied","Data":"9491c650a97a6ac3253b23c768772b144f10e50081340ccd666d88d9bbfbcf6b"}
Oct 31 02:10:40.098844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.097718    5293 scope.go:122] "RemoveContainer" containerID="ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b"
Oct 31 02:10:40.125627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.125607    5293 scope.go:122] "RemoveContainer" containerID="ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b"
Oct 31 02:10:40.128282 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:10:40.128244    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b\": container with ID starting with ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b not found: ID does not exist" containerID="ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b"
Oct 31 02:10:40.128423 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.128294    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b"} err="failed to get container status \"ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b\": rpc error: code = NotFound desc = could not find container \"ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b\": container with ID starting with ecffd423c9ea5f6468633ff8764f85669b4d55f6e0b8f2c172588da32933421b not found: ID does not exist"
Oct 31 02:10:40.133209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.133166    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"]
Oct 31 02:10:40.138793 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.138768    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-lh8rk"]
Oct 31 02:10:40.178763 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.178732    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc6ec04-1cc3-4859-a53b-10fc82ff3433" path="/var/lib/kubelet/pods/0cc6ec04-1cc3-4859-a53b-10fc82ff3433/volumes"
Oct 31 02:10:40.202916 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.202892    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:40.504009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.503984    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"]
Oct 31 02:10:40.504183 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:10:40.504139    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18654831_3ecb_46ae_bd66_59c4725008fe.slice/crio-44cdb7e46bc00e82d4ccb2f9ee61d01254447433aaa5272560ded17bf0f4a7aa WatchSource:0}: Error finding container 44cdb7e46bc00e82d4ccb2f9ee61d01254447433aaa5272560ded17bf0f4a7aa: Status 404 returned error can't find the container with id 44cdb7e46bc00e82d4ccb2f9ee61d01254447433aaa5272560ded17bf0f4a7aa
Oct 31 02:10:40.505633 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:40.505612    5293 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:10:41.111359 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:41.111326    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8" event={"ID":"18654831-3ecb-46ae-bd66-59c4725008fe","Type":"ContainerStarted","Data":"44cdb7e46bc00e82d4ccb2f9ee61d01254447433aaa5272560ded17bf0f4a7aa"}
Oct 31 02:10:41.783427 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:10:41.783380    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:10:42.125585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:42.125508    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8" event={"ID":"18654831-3ecb-46ae-bd66-59c4725008fe","Type":"ContainerStarted","Data":"1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80"}
Oct 31 02:10:42.125996 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:42.125638    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:42.152243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:42.152194    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8" podStartSLOduration=299.628796152 podStartE2EDuration="5m0.152179879s" totalImagesPullingTime="523.383727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:42.145586585 +0000 UTC m=+3000.094234980" watchObservedRunningTime="2026-10-31 02:10:42.152179879 +0000 UTC m=+3000.100828270"
Oct 31 02:10:43.035300 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:43.035269    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"]
Oct 31 02:10:43.042054 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:43.042006    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-895d3754-vsx97"]
Oct 31 02:10:44.162757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.162732    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 02:10:44.169169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.169142    5293 generic.go:356] "Generic (PLEG): container finished" podID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerID="db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088" exitCode=0
Oct 31 02:10:44.169308 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.169282    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"
Oct 31 02:10:44.180196 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.180173    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33e958f-e08e-4cbe-920d-188ff8a89b23" path="/var/lib/kubelet/pods/a33e958f-e08e-4cbe-920d-188ff8a89b23/volumes"
Oct 31 02:10:44.182711 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.182688    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" event={"ID":"177634d2-0f59-400b-8ad4-77188f8ce3cf","Type":"ContainerDied","Data":"db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088"}
Oct 31 02:10:44.182814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.182749    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n" event={"ID":"177634d2-0f59-400b-8ad4-77188f8ce3cf","Type":"ContainerDied","Data":"82361b2cb109db4d34a2773cf4dcab53befdcfeaf6d92d28f01d6221fc8bfb63"}
Oct 31 02:10:44.182814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.182780    5293 scope.go:122] "RemoveContainer" containerID="db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088"
Oct 31 02:10:44.200594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.200566    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"]
Oct 31 02:10:44.200985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.200965    5293 scope.go:122] "RemoveContainer" containerID="823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686"
Oct 31 02:10:44.201602 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.201576    5293 cpu_manager.go:414] "containerMap: removing container" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy"
Oct 31 02:10:44.201602 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.201598    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy"
Oct 31 02:10:44.201783 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.201772    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy"
Oct 31 02:10:44.205053 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205008    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177634d2-0f59-400b-8ad4-77188f8ce3cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177634d2-0f59-400b-8ad4-77188f8ce3cf-istiod-ca-cert\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.205175 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205107    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-data\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.205260 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205239    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-certs\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.205645 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205336    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-token\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-token\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.205645 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205440    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-podinfo\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.205645 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205592    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-envoy\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.205645 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205609    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-data" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.205645 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205625    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-socket\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.206310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205630    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-certs" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.206310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205694    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-credential-socket\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.206310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205770    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-kube-api-access-fzc59\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-kube-api-access-fzc59\") pod \"177634d2-0f59-400b-8ad4-77188f8ce3cf\" (UID: \"177634d2-0f59-400b-8ad4-77188f8ce3cf\") "
Oct 31 02:10:44.206310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.205985    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-socket" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.206310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.206092    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-credential-socket" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.207160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.207140    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177634d2-0f59-400b-8ad4-77188f8ce3cf-istiod-ca-cert" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:44.207677 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.207644    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.208204 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.208178    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.208305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.208209    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.208305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.208225    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.208305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.208239    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177634d2-0f59-400b-8ad4-77188f8ce3cf-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.208305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.208252    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.216063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.213350    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f3e2ace89eb9\""
Oct 31 02:10:44.216063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.213963    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-envoy" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.216063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.214079    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-415db4c48eaf\""
Oct 31 02:10:44.216063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.214701    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7475ba9a0297\""
Oct 31 02:10:44.216867 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.216840    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-kube-api-access-fzc59" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "kube-api-access-fzc59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:44.218872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.218847    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-podinfo" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:10:44.223074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.223030    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"]
Oct 31 02:10:44.224683 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.224369    5293 scope.go:122] "RemoveContainer" containerID="db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088"
Oct 31 02:10:44.225556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.225525    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-token" pod "177634d2-0f59-400b-8ad4-77188f8ce3cf" (UID: "177634d2-0f59-400b-8ad4-77188f8ce3cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:44.227278 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:10:44.227245    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088\": container with ID starting with db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088 not found: ID does not exist" containerID="db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088"
Oct 31 02:10:44.227390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.227289    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088"} err="failed to get container status \"db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088\": rpc error: code = NotFound desc = could not find container \"db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088\": container with ID starting with db8d890a4a38dfd176239f5a879b40d164513cd1d09b04e5927496027a21d088 not found: ID does not exist"
Oct 31 02:10:44.227390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.227313    5293 scope.go:122] "RemoveContainer" containerID="823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686"
Oct 31 02:10:44.230340 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:10:44.230138    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686\": container with ID starting with 823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686 not found: ID does not exist" containerID="823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686"
Oct 31 02:10:44.230340 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.230177    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686"} err="failed to get container status \"823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686\": rpc error: code = NotFound desc = could not find container \"823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686\": container with ID starting with 823564a9597da8a11e35ee82caa7c93512d5f98c133a63671a6af6e6658ee686 not found: ID does not exist"
Oct 31 02:10:44.321352 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.321311    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-plugins-config-volume\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.321650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.321627    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.321791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.321746    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6j2\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-kube-api-access-tf6j2\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.321948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.321815    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-routing-overlay-volume\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.321948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.321939    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.322167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.321952    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.322167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.321962    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177634d2-0f59-400b-8ad4-77188f8ce3cf-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.322167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.321970    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzc59\" (UniqueName: \"kubernetes.io/projected/177634d2-0f59-400b-8ad4-77188f8ce3cf-kube-api-access-fzc59\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.423272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.423224    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-routing-overlay-volume\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.423474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.423329    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-plugins-config-volume\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.423644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.423627    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.423812 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.423794    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6j2\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-kube-api-access-tf6j2\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.424126 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.424097    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-routing-overlay-volume\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.424271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.424246    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-plugins-config-volume\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.426598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.426575    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.435958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.435935    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6j2\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-kube-api-access-tf6j2\") pod \"payload-processing-external-model-57745f7bf-vqzk8\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") " pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:44.496314 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.496281    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"]
Oct 31 02:10:44.503540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.503474    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-z285n"]
Oct 31 02:10:44.574251 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:44.574223    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:45.071342 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:10:45.071307    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b073a6_5030_4f71_b024_b2d0c436051b.slice/crio-f64377c749bb717f18035c09b86c259270882ad81e11a1219521d3c68c6fb540 WatchSource:0}: Error finding container f64377c749bb717f18035c09b86c259270882ad81e11a1219521d3c68c6fb540: Status 404 returned error can't find the container with id f64377c749bb717f18035c09b86c259270882ad81e11a1219521d3c68c6fb540
Oct 31 02:10:45.071467 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:45.071380    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"]
Oct 31 02:10:45.169131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:45.169073    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"]
Oct 31 02:10:45.169513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:45.169295    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy" containerID="cri-o://0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc" gracePeriod=30
Oct 31 02:10:45.188225 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:45.188187    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8" event={"ID":"41b073a6-5030-4f71-b024-b2d0c436051b","Type":"ContainerStarted","Data":"6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514"}
Oct 31 02:10:45.188398 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:45.188251    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8" event={"ID":"41b073a6-5030-4f71-b024-b2d0c436051b","Type":"ContainerStarted","Data":"f64377c749bb717f18035c09b86c259270882ad81e11a1219521d3c68c6fb540"}
Oct 31 02:10:45.188398 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:45.188358    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:45.230359 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:45.230293    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8" podStartSLOduration=38.230269371 podStartE2EDuration="38.230269371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:45.211736722 +0000 UTC m=+3003.160385121" watchObservedRunningTime="2026-10-31 02:10:45.230269371 +0000 UTC m=+3003.178917764"
Oct 31 02:10:46.177738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:46.177706    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" path="/var/lib/kubelet/pods/177634d2-0f59-400b-8ad4-77188f8ce3cf/volumes"
Oct 31 02:10:48.191816 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:48.191527    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:10:48.920075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:48.919577    5293 scope.go:122] "RemoveContainer" containerID="456d57683fd96c2b2762c0e43cc655a6470d86d80069fc55b242e3fd83a03fb7"
Oct 31 02:10:48.950071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:48.941096    5293 scope.go:122] "RemoveContainer" containerID="fa5adc20eea9ccbd155114a9c0f6c318dd7d0b64d1e5923430d77c124c2f3984"
Oct 31 02:10:48.971419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:48.970892    5293 scope.go:122] "RemoveContainer" containerID="86262cae1db54a5d788d5daa4bcd3087dc2edc96c14a985e35bbb65b9861079c"
Oct 31 02:10:50.564178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.564148    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 02:10:50.600418 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.600393    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"]
Oct 31 02:10:50.601272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.601246    5293 cpu_manager.go:414] "containerMap: removing container" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy"
Oct 31 02:10:50.601272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.601265    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy"
Oct 31 02:10:50.601476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.601311    5293 cpu_manager.go:414] "containerMap: removing container" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy"
Oct 31 02:10:50.601476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.601317    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy"
Oct 31 02:10:50.601476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.601471    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="177634d2-0f59-400b-8ad4-77188f8ce3cf" containerName="istio-proxy"
Oct 31 02:10:50.601637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.601482    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a20441-02fd-4989-b638-2bde95779d23" containerName="istio-proxy"
Oct 31 02:10:50.605466 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.605446    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.628055 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.628010    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"]
Oct 31 02:10:50.679990 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.679957    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-credential-socket\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680022    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-certs\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680174    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-data\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680267    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-credential-socket" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:50.680405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680346    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-certs" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:50.680513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680403    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a20441-02fd-4989-b638-2bde95779d23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91a20441-02fd-4989-b638-2bde95779d23-istiod-ca-cert\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680447    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-data" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:50.680513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680489    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-istio-token\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-istio-token\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680680 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680561    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-socket\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680680 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680606    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-kube-api-access-2h9z4\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-kube-api-access-2h9z4\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680680 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680643    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-envoy\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680708    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91a20441-02fd-4989-b638-2bde95779d23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91a20441-02fd-4989-b638-2bde95779d23-istio-podinfo\") pod \"91a20441-02fd-4989-b638-2bde95779d23\" (UID: \"91a20441-02fd-4989-b638-2bde95779d23\") "
Oct 31 02:10:50.680856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.680818    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a20441-02fd-4989-b638-2bde95779d23-istiod-ca-cert" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:50.681108 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.681081    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-socket" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:50.681208 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.681102    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.681208 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.681142    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.681357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.681339    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.681471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.681448    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36f3ee8-e967-4465-a485-7a9b040e02c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.681917 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.681892    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.682240 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682147    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.682348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682323    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmsb9\" (UniqueName: \"kubernetes.io/projected/a36f3ee8-e967-4465-a485-7a9b040e02c2-kube-api-access-jmsb9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.682420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682403    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.682707 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682637    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.682889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682868    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.682965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682895    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91a20441-02fd-4989-b638-2bde95779d23-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.682965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682911    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.682965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682926    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.682965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.682939    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.683653 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.683628    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-istio-token" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:50.684265 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.684240    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-envoy" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:50.684703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.684683    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-kube-api-access-2h9z4" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "kube-api-access-2h9z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:50.684778 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.684714    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91a20441-02fd-4989-b638-2bde95779d23-istio-podinfo" pod "91a20441-02fd-4989-b638-2bde95779d23" (UID: "91a20441-02fd-4989-b638-2bde95779d23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:10:50.784245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784163    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784201    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784342    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784377    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36f3ee8-e967-4465-a485-7a9b040e02c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784525    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784562    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784565    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784696    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmsb9\" (UniqueName: \"kubernetes.io/projected/a36f3ee8-e967-4465-a485-7a9b040e02c2-kube-api-access-jmsb9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784735    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784848    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.784896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.784870    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.785156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.785074    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.785156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.785098    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9z4\" (UniqueName: \"kubernetes.io/projected/91a20441-02fd-4989-b638-2bde95779d23-kube-api-access-2h9z4\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.785156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.785113    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91a20441-02fd-4989-b638-2bde95779d23-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.785156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.785127    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91a20441-02fd-4989-b638-2bde95779d23-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.785376 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.785188    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.785376 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.785198    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.785376 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.785236    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36f3ee8-e967-4465-a485-7a9b040e02c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.786948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.786925    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.787261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.787242    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.796077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.796026    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36f3ee8-e967-4465-a485-7a9b040e02c2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.796356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.796337    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmsb9\" (UniqueName: \"kubernetes.io/projected/a36f3ee8-e967-4465-a485-7a9b040e02c2-kube-api-access-jmsb9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x47hr\" (UID: \"a36f3ee8-e967-4465-a485-7a9b040e02c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:50.949181 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:50.948883    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:51.312267 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:10:51.312237    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36f3ee8_e967_4465_a485_7a9b040e02c2.slice/crio-e844db60bf269fef29f2b2046cd4ef632139bfc9c6f7368561a688003bc4babd WatchSource:0}: Error finding container e844db60bf269fef29f2b2046cd4ef632139bfc9c6f7368561a688003bc4babd: Status 404 returned error can't find the container with id e844db60bf269fef29f2b2046cd4ef632139bfc9c6f7368561a688003bc4babd
Oct 31 02:10:51.313087 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.313062    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"]
Oct 31 02:10:51.316510 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.316469    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:10:51.316615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.316561    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:10:51.316615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.316603    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:10:51.341372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.341338    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" event={"ID":"a36f3ee8-e967-4465-a485-7a9b040e02c2","Type":"ContainerStarted","Data":"e844db60bf269fef29f2b2046cd4ef632139bfc9c6f7368561a688003bc4babd"}
Oct 31 02:10:51.344489 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.344463    5293 generic.go:356] "Generic (PLEG): container finished" podID="91a20441-02fd-4989-b638-2bde95779d23" containerID="0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc" exitCode=0
Oct 31 02:10:51.344635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.344587    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" event={"ID":"91a20441-02fd-4989-b638-2bde95779d23","Type":"ContainerDied","Data":"0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc"}
Oct 31 02:10:51.344635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.344630    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk" event={"ID":"91a20441-02fd-4989-b638-2bde95779d23","Type":"ContainerDied","Data":"c151df719a6a675f0dcab9de644d522b24dd6309182bb401a813f7f8d5e7d319"}
Oct 31 02:10:51.344802 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.344660    5293 scope.go:122] "RemoveContainer" containerID="0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc"
Oct 31 02:10:51.344864 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.344844    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"
Oct 31 02:10:51.364763 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.364589    5293 scope.go:122] "RemoveContainer" containerID="0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc"
Oct 31 02:10:51.364880 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:10:51.364859    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc\": container with ID starting with 0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc not found: ID does not exist" containerID="0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc"
Oct 31 02:10:51.364938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.364892    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc"} err="failed to get container status \"0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc\": rpc error: code = NotFound desc = could not find container \"0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc\": container with ID starting with 0faf93d9f084831ad8b0b3c75a2743bf2e717e080ae125354ada82be152783bc not found: ID does not exist"
Oct 31 02:10:51.393266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.393241    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"]
Oct 31 02:10:51.400621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:51.400599    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-2k7lk"]
Oct 31 02:10:52.177249 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:52.177217    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a20441-02fd-4989-b638-2bde95779d23" path="/var/lib/kubelet/pods/91a20441-02fd-4989-b638-2bde95779d23/volumes"
Oct 31 02:10:52.356138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:52.356089    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" event={"ID":"a36f3ee8-e967-4465-a485-7a9b040e02c2","Type":"ContainerStarted","Data":"fce4fd0023c877b4abc326dac594dfd5338379d8b679d9f60376144cd6c61bf6"}
Oct 31 02:10:52.382528 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:52.382474    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podStartSLOduration=319.382458233 podStartE2EDuration="5m19.382458233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:52.380031835 +0000 UTC m=+3010.328680252" watchObservedRunningTime="2026-10-31 02:10:52.382458233 +0000 UTC m=+3010.331106624"
Oct 31 02:10:52.950056 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:52.950021    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:10:52.951966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:52.951940    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 31 02:10:52.952132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:52.951987    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 31 02:10:53.964726 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:53.951503    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 31 02:10:53.964726 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:53.951565    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 31 02:10:54.952544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:54.952197    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 31 02:10:54.952544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:54.952277    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 31 02:10:55.949970 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:55.949938    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 31 02:10:55.950403 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:55.950006    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 31 02:10:56.232615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:56.232526    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:10:56.463066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:56.445420    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"]
Oct 31 02:10:56.640316 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:56.640123    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:10:56.956501 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:56.956463    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 31 02:10:56.957183 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:56.957139    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 31 02:10:57.958093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:57.956592    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 31 02:10:57.958093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:57.956656    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 31 02:10:58.949753 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:58.949714    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 31 02:10:58.949933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:58.949783    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 31 02:10:59.950441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:59.950401    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 31 02:10:59.951116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:10:59.951080    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 31 02:11:01.450184 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:01.450120    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b" podUID="f4c5fe57-9b80-4679-a263-2515de1eef94" containerName="payload-processing" containerID="cri-o://31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7" gracePeriod=25
Oct 31 02:11:01.993217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:01.993171    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:11:01.993425 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:01.993232    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:11:02.516668 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.516641    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:11:02.595383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.595343    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-plugins-config-volume\") pod \"f4c5fe57-9b80-4679-a263-2515de1eef94\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") "
Oct 31 02:11:02.595587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.595455    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-kube-api-access-f2hz2\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-kube-api-access-f2hz2\") pod \"f4c5fe57-9b80-4679-a263-2515de1eef94\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") "
Oct 31 02:11:02.595587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.595482    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-provider-credentials\") pod \"f4c5fe57-9b80-4679-a263-2515de1eef94\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") "
Oct 31 02:11:02.595727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.595600    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-routing-overlay-volume\") pod \"f4c5fe57-9b80-4679-a263-2515de1eef94\" (UID: \"f4c5fe57-9b80-4679-a263-2515de1eef94\") "
Oct 31 02:11:02.596890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.596860    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-routing-overlay-volume" pod "f4c5fe57-9b80-4679-a263-2515de1eef94" (UID: "f4c5fe57-9b80-4679-a263-2515de1eef94"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:02.600464 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.600266    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-provider-credentials" pod "f4c5fe57-9b80-4679-a263-2515de1eef94" (UID: "f4c5fe57-9b80-4679-a263-2515de1eef94"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:02.602473 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.600767    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-plugins-config-volume" pod "f4c5fe57-9b80-4679-a263-2515de1eef94" (UID: "f4c5fe57-9b80-4679-a263-2515de1eef94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:02.605501 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.605472    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-kube-api-access-f2hz2" pod "f4c5fe57-9b80-4679-a263-2515de1eef94" (UID: "f4c5fe57-9b80-4679-a263-2515de1eef94"). InnerVolumeSpecName "kube-api-access-f2hz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:02.607912 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.607883    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"]
Oct 31 02:11:02.609409 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.609383    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f4c5fe57-9b80-4679-a263-2515de1eef94" containerName="payload-processing"
Oct 31 02:11:02.609517 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.609411    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c5fe57-9b80-4679-a263-2515de1eef94" containerName="payload-processing"
Oct 31 02:11:02.609702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.609683    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c5fe57-9b80-4679-a263-2515de1eef94" containerName="payload-processing"
Oct 31 02:11:02.615341 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.615318    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.616208 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.616187    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"]
Oct 31 02:11:02.697829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.697774    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlk8x\" (UniqueName: \"kubernetes.io/projected/0cef45b4-73c6-463c-a1e0-716629cc73b0-kube-api-access-vlk8x\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698024 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.697904    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698024 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.697938    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698172 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698111    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698313    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cef45b4-73c6-463c-a1e0-716629cc73b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698415    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698518    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698662 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698568    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698710 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698680    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.698941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698921    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:02.698988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698950    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c5fe57-9b80-4679-a263-2515de1eef94-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:02.698988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698966    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2hz2\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-kube-api-access-f2hz2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:02.698988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.698979    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c5fe57-9b80-4679-a263-2515de1eef94-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:02.770676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.770563    5293 generic.go:356] "Generic (PLEG): container finished" podID="f4c5fe57-9b80-4679-a263-2515de1eef94" containerID="31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7" exitCode=0
Oct 31 02:11:02.770676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.770616    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b" event={"ID":"f4c5fe57-9b80-4679-a263-2515de1eef94","Type":"ContainerDied","Data":"31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7"}
Oct 31 02:11:02.770676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.770637    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"
Oct 31 02:11:02.770676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.770693    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b" event={"ID":"f4c5fe57-9b80-4679-a263-2515de1eef94","Type":"ContainerDied","Data":"85366bdd894dd726bc61e093e1bd41cdb80aafb54baa79d5cef88b7e70ad3eaf"}
Oct 31 02:11:02.771155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.770729    5293 scope.go:122] "RemoveContainer" containerID="31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7"
Oct 31 02:11:02.788471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.788186    5293 scope.go:122] "RemoveContainer" containerID="31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7"
Oct 31 02:11:02.790006 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:11:02.788924    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7\": container with ID starting with 31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7 not found: ID does not exist" containerID="31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7"
Oct 31 02:11:02.790006 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.788965    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7"} err="failed to get container status \"31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7\": rpc error: code = NotFound desc = could not find container \"31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7\": container with ID starting with 31a26bf35bdcd0fcf038b9cf4c77c4e83e8d42a00cfa1438784077ba5875b8b7 not found: ID does not exist"
Oct 31 02:11:02.800735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.800710    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.800866 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.800851    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cef45b4-73c6-463c-a1e0-716629cc73b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.800943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.800925    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801014    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801077    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801152    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801293    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlk8x\" (UniqueName: \"kubernetes.io/projected/0cef45b4-73c6-463c-a1e0-716629cc73b0-kube-api-access-vlk8x\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801355    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801384    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801440    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801453    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801944 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801675    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cef45b4-73c6-463c-a1e0-716629cc73b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801944 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801807    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.801944 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.801900    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.804709 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.804684    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"]
Oct 31 02:11:02.805009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.804981    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.805189 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.805167    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.810187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.810167    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c8bffbccc-lfg9b"]
Oct 31 02:11:02.812745 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.812719    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cef45b4-73c6-463c-a1e0-716629cc73b0-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.813560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.813536    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlk8x\" (UniqueName: \"kubernetes.io/projected/0cef45b4-73c6-463c-a1e0-716629cc73b0-kube-api-access-vlk8x\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d8xb2\" (UID: \"0cef45b4-73c6-463c-a1e0-716629cc73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:02.937222 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:02.937195    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:03.027119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.026169    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:11:03.027119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.026236    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:11:03.394747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.394571    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"]
Oct 31 02:11:03.404895 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.404856    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:11:03.405029 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.404945    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:11:03.405029 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.404993    5293 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 02:11:03.811236 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.811195    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" event={"ID":"0cef45b4-73c6-463c-a1e0-716629cc73b0","Type":"ContainerStarted","Data":"2c4e242518e8a2217b184bf9f1f174c6c58ee8f1271fc9eeb7b8f37fe7b6fcc5"}
Oct 31 02:11:03.811746 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.811268    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" event={"ID":"0cef45b4-73c6-463c-a1e0-716629cc73b0","Type":"ContainerStarted","Data":"6e1ba0adb80bdcc29dcf9e60883c446fc978a63391a0da67c5b64be0f81bb265"}
Oct 31 02:11:03.845027 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.844963    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podStartSLOduration=330.844943903 podStartE2EDuration="5m30.844943903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:11:03.840225822 +0000 UTC m=+3021.788874241" watchObservedRunningTime="2026-10-31 02:11:03.844943903 +0000 UTC m=+3021.793592298"
Oct 31 02:11:03.938355 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:03.938328    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:04.067237 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:04.067151    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:11:04.067237 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:04.067202    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:11:04.177799 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:04.177764    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c5fe57-9b80-4679-a263-2515de1eef94" path="/var/lib/kubelet/pods/f4c5fe57-9b80-4679-a263-2515de1eef94/volumes"
Oct 31 02:11:04.941415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:04.940878    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 31 02:11:04.941415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:04.940941    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podUID="0cef45b4-73c6-463c-a1e0-716629cc73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 31 02:11:05.115222 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:05.115189    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x47hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:11:05.115501 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:05.115477    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr" podUID="a36f3ee8-e967-4465-a485-7a9b040e02c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:11:05.639569 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:05.637227    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:11:05.640671 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:05.640499    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:11:05.678740 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:05.678708    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x47hr"
Oct 31 02:11:05.995069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:05.948255    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 31 02:11:05.995069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:05.948325    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podUID="0cef45b4-73c6-463c-a1e0-716629cc73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 31 02:11:06.938552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:06.938498    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 31 02:11:06.938769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:06.938562    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podUID="0cef45b4-73c6-463c-a1e0-716629cc73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 31 02:11:07.938637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:07.938601    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 31 02:11:07.939230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:07.939198    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podUID="0cef45b4-73c6-463c-a1e0-716629cc73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 31 02:11:08.938305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:08.938268    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 31 02:11:08.938594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:08.938564    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podUID="0cef45b4-73c6-463c-a1e0-716629cc73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 31 02:11:09.938488 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:09.938453    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 31 02:11:09.939023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:09.938575    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podUID="0cef45b4-73c6-463c-a1e0-716629cc73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 31 02:11:10.937986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:10.937948    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 31 02:11:10.938206 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:10.938033    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podUID="0cef45b4-73c6-463c-a1e0-716629cc73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 31 02:11:11.781017 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:11:11.780976    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:11:11.940733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:11.940621    5293 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 31 02:11:11.941013 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:11.940983    5293 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2" podUID="0cef45b4-73c6-463c-a1e0-716629cc73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 31 02:11:13.214379 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:13.214347    5293 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:13.214878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:13.214694    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:13.216892 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:13.216859    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d8xb2"
Oct 31 02:11:35.846887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:35.846849    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"]
Oct 31 02:11:35.853550 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:35.853529    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:35.859389 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:35.859363    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:11:35.859612 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:35.859454    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:11:35.861483 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:35.861459    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"]
Oct 31 02:11:35.916146 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:35.916101    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:35.916342 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:35.916257    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:35.916456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:35.916440    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fflm5\" (UniqueName: \"kubernetes.io/projected/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-kube-api-access-fflm5\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:36.018999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.018960    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fflm5\" (UniqueName: \"kubernetes.io/projected/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-kube-api-access-fflm5\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:36.019296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.019278    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:36.019452 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.019435    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:36.020205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.020138    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:36.021929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.021907    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:36.030669 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.030645    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fflm5\" (UniqueName: \"kubernetes.io/projected/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-kube-api-access-fflm5\") pod \"maas-api-revoke-sub-564e1078-4xc57\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:36.171504 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.171477    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:36.300006 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.299868    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:11:36.493326 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.493298    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"]
Oct 31 02:11:36.694937 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:36.694904    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57" event={"ID":"eb2ecbb7-d38b-4115-8a8b-48663b9c733a","Type":"ContainerStarted","Data":"2f3259696967a1a0dc479e696c5b38ed2b220a80ff64d5be29aa3654cfd7f977"}
Oct 31 02:11:37.786292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:37.778088    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57" event={"ID":"eb2ecbb7-d38b-4115-8a8b-48663b9c733a","Type":"ContainerStarted","Data":"add01916dcc5daa07bed8f17fc963c09e9d455dc98bfec5a687d014416b20c6b"}
Oct 31 02:11:37.831661 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:37.830300    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57" podStartSLOduration=2.830279049 podStartE2EDuration="2.830279049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:11:37.821437655 +0000 UTC m=+3055.770086050" watchObservedRunningTime="2026-10-31 02:11:37.830279049 +0000 UTC m=+3055.778927422"
Oct 31 02:11:38.788390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:38.788352    5293 generic.go:356] "Generic (PLEG): container finished" podID="eb2ecbb7-d38b-4115-8a8b-48663b9c733a" containerID="add01916dcc5daa07bed8f17fc963c09e9d455dc98bfec5a687d014416b20c6b" exitCode=0
Oct 31 02:11:38.788878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:38.788462    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57" event={"ID":"eb2ecbb7-d38b-4115-8a8b-48663b9c733a","Type":"ContainerDied","Data":"add01916dcc5daa07bed8f17fc963c09e9d455dc98bfec5a687d014416b20c6b"}
Oct 31 02:11:40.176939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.176910    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:40.280285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.280248    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-xks-ca\") pod \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") "
Oct 31 02:11:40.280492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.280368    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-openshift-ca\") pod \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") "
Oct 31 02:11:40.280688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.280670    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-kube-api-access-fflm5\" (UniqueName: \"kubernetes.io/projected/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-kube-api-access-fflm5\") pod \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\" (UID: \"eb2ecbb7-d38b-4115-8a8b-48663b9c733a\") "
Oct 31 02:11:40.281277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.281240    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-openshift-ca" pod "eb2ecbb7-d38b-4115-8a8b-48663b9c733a" (UID: "eb2ecbb7-d38b-4115-8a8b-48663b9c733a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:40.282668 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.282636    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:40.283666 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.283641    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-xks-ca" pod "eb2ecbb7-d38b-4115-8a8b-48663b9c733a" (UID: "eb2ecbb7-d38b-4115-8a8b-48663b9c733a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:40.284404 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.284375    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-kube-api-access-fflm5" pod "eb2ecbb7-d38b-4115-8a8b-48663b9c733a" (UID: "eb2ecbb7-d38b-4115-8a8b-48663b9c733a"). InnerVolumeSpecName "kube-api-access-fflm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:40.387114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.386320    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fflm5\" (UniqueName: \"kubernetes.io/projected/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-kube-api-access-fflm5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:40.387114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.386498    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2ecbb7-d38b-4115-8a8b-48663b9c733a-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:40.810500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.810467    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"
Oct 31 02:11:40.810702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.810505    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57" event={"ID":"eb2ecbb7-d38b-4115-8a8b-48663b9c733a","Type":"ContainerDied","Data":"2f3259696967a1a0dc479e696c5b38ed2b220a80ff64d5be29aa3654cfd7f977"}
Oct 31 02:11:40.810702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:40.810569    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f3259696967a1a0dc479e696c5b38ed2b220a80ff64d5be29aa3654cfd7f977"
Oct 31 02:11:41.781877 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:11:41.781830    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:11:48.812979 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.812937    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"]
Oct 31 02:11:48.813878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.813852    5293 cpu_manager.go:414] "containerMap: removing container" podUID="eb2ecbb7-d38b-4115-8a8b-48663b9c733a" containerName="revoke-keys"
Oct 31 02:11:48.813878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.813875    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb2ecbb7-d38b-4115-8a8b-48663b9c733a" containerName="revoke-keys"
Oct 31 02:11:48.814034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.814029    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb2ecbb7-d38b-4115-8a8b-48663b9c733a" containerName="revoke-keys"
Oct 31 02:11:48.818223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.818204    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:48.821763 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.821345    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:11:48.822009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.821977    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:11:48.829628 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.829377    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"]
Oct 31 02:11:48.916103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.913560    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2j2b\" (UniqueName: \"kubernetes.io/projected/0171f979-aedd-4d20-873c-c43ab02af336-kube-api-access-b2j2b\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:48.916103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.913683    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:48.916103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:48.913935    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:49.027794 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.027750    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:49.028187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.028056    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2j2b\" (UniqueName: \"kubernetes.io/projected/0171f979-aedd-4d20-873c-c43ab02af336-kube-api-access-b2j2b\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:49.028187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.028144    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:49.029168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.029087    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:49.044902 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.044787    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:49.071117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.069890    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2j2b\" (UniqueName: \"kubernetes.io/projected/0171f979-aedd-4d20-873c-c43ab02af336-kube-api-access-b2j2b\") pod \"maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:49.093533 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.093497    5293 scope.go:122] "RemoveContainer" containerID="1aaa211dd5f043c5b151b889bdb9948432b09994f5f178f6b3c3d51e98249f60"
Oct 31 02:11:49.106892 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.106867    5293 scope.go:122] "RemoveContainer" containerID="54132afb1faad0ad2b599b6a48e35a8c6051815531786ac44df03d4f816e0b49"
Oct 31 02:11:49.126456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.126432    5293 scope.go:122] "RemoveContainer" containerID="47b12752f90c633044d5d7a16736d773d00ba3c274bed1f7e00dbb36e0d6d344"
Oct 31 02:11:49.139714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.139690    5293 scope.go:122] "RemoveContainer" containerID="c5b3c23452abee11c71a2c9dd39f1cc4c73cb091a7f835d57cddd81949f0a855"
Oct 31 02:11:49.146110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.146032    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:49.452315 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:11:49.452287    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0171f979_aedd_4d20_873c_c43ab02af336.slice/crio-61a05ddc36db4ed8d4397cb232851ae03496695ca4252505865949a299b72c5c WatchSource:0}: Error finding container 61a05ddc36db4ed8d4397cb232851ae03496695ca4252505865949a299b72c5c: Status 404 returned error can't find the container with id 61a05ddc36db4ed8d4397cb232851ae03496695ca4252505865949a299b72c5c
Oct 31 02:11:49.453136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:49.453114    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"]
Oct 31 02:11:50.109243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:50.109146    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc" event={"ID":"0171f979-aedd-4d20-873c-c43ab02af336","Type":"ContainerStarted","Data":"38a4d5ec763a8826708ca97e875e6c31ef0a7dd6b6483e52d03f0e1f3004d805"}
Oct 31 02:11:50.109243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:50.109210    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc" event={"ID":"0171f979-aedd-4d20-873c-c43ab02af336","Type":"ContainerStarted","Data":"61a05ddc36db4ed8d4397cb232851ae03496695ca4252505865949a299b72c5c"}
Oct 31 02:11:50.129316 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:50.129031    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc" podStartSLOduration=2.129015419 podStartE2EDuration="2.129015419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:11:50.127986982 +0000 UTC m=+3068.076635375" watchObservedRunningTime="2026-10-31 02:11:50.129015419 +0000 UTC m=+3068.077663815"
Oct 31 02:11:51.178277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:51.178244    5293 generic.go:356] "Generic (PLEG): container finished" podID="0171f979-aedd-4d20-873c-c43ab02af336" containerID="38a4d5ec763a8826708ca97e875e6c31ef0a7dd6b6483e52d03f0e1f3004d805" exitCode=0
Oct 31 02:11:51.178853 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:51.178832    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc" event={"ID":"0171f979-aedd-4d20-873c-c43ab02af336","Type":"ContainerDied","Data":"38a4d5ec763a8826708ca97e875e6c31ef0a7dd6b6483e52d03f0e1f3004d805"}
Oct 31 02:11:52.447380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.447209    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:52.555357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.555007    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-xks-ca\") pod \"0171f979-aedd-4d20-873c-c43ab02af336\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") "
Oct 31 02:11:52.555357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.555209    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-openshift-ca\") pod \"0171f979-aedd-4d20-873c-c43ab02af336\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") "
Oct 31 02:11:52.555357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.555321    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0171f979-aedd-4d20-873c-c43ab02af336-kube-api-access-b2j2b\" (UniqueName: \"kubernetes.io/projected/0171f979-aedd-4d20-873c-c43ab02af336-kube-api-access-b2j2b\") pod \"0171f979-aedd-4d20-873c-c43ab02af336\" (UID: \"0171f979-aedd-4d20-873c-c43ab02af336\") "
Oct 31 02:11:52.561482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.559281    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-openshift-ca" pod "0171f979-aedd-4d20-873c-c43ab02af336" (UID: "0171f979-aedd-4d20-873c-c43ab02af336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:52.569988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.569962    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:52.584456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.584428    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-xks-ca" pod "0171f979-aedd-4d20-873c-c43ab02af336" (UID: "0171f979-aedd-4d20-873c-c43ab02af336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:52.586214 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.586188    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0171f979-aedd-4d20-873c-c43ab02af336-kube-api-access-b2j2b" pod "0171f979-aedd-4d20-873c-c43ab02af336" (UID: "0171f979-aedd-4d20-873c-c43ab02af336"). InnerVolumeSpecName "kube-api-access-b2j2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:52.672714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.672677    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0171f979-aedd-4d20-873c-c43ab02af336-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:52.672714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:52.672708    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2j2b\" (UniqueName: \"kubernetes.io/projected/0171f979-aedd-4d20-873c-c43ab02af336-kube-api-access-b2j2b\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:11:53.288319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:53.288284    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc" event={"ID":"0171f979-aedd-4d20-873c-c43ab02af336","Type":"ContainerDied","Data":"61a05ddc36db4ed8d4397cb232851ae03496695ca4252505865949a299b72c5c"}
Oct 31 02:11:53.288582 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:53.288565    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61a05ddc36db4ed8d4397cb232851ae03496695ca4252505865949a299b72c5c"
Oct 31 02:11:53.288774 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:53.288762    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"
Oct 31 02:11:54.033016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:54.032984    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"]
Oct 31 02:11:54.051182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:54.051145    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a5c53b-mntwh"]
Oct 31 02:11:54.176628 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:54.176594    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6540f6-e146-40ea-9227-3a9aa21bf5d2" path="/var/lib/kubelet/pods/1f6540f6-e146-40ea-9227-3a9aa21bf5d2/volumes"
Oct 31 02:11:59.079064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:59.078379    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"]
Oct 31 02:11:59.356837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:59.355794    5293 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" secret="" err="secret \"payload-processing-e2e-extmod-a-2186c3-dockercfg-t4vjw\" not found"
Oct 31 02:11:59.408259 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:11:59.408232    5293 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-extmod-a-2186c3: configmap "payload-processing-plugins-e2e-extmod-a-2186c3" not found
Oct 31 02:11:59.408598 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:11:59.408572    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume podName:b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa nodeName:}" failed. No retries permitted until 2026-10-31 02:11:59.90853922 +0000 UTC m=+3077.857187619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume") pod "payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" (UID: "b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa") : configmap "payload-processing-plugins-e2e-extmod-a-2186c3" not found
Oct 31 02:11:59.901345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:11:59.894130    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"]
Oct 31 02:11:59.962648 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:11:59.962610    5293 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-extmod-a-2186c3: configmap "payload-processing-plugins-e2e-extmod-a-2186c3" not found
Oct 31 02:11:59.962820 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:11:59.962692    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume podName:b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa nodeName:}" failed. No retries permitted until 2026-10-31 02:12:00.962671709 +0000 UTC m=+3078.911320085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume") pod "payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" (UID: "b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa") : configmap "payload-processing-plugins-e2e-extmod-a-2186c3" not found
Oct 31 02:12:01.039788 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:01.039749    5293 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-extmod-a-2186c3: configmap "payload-processing-plugins-e2e-extmod-a-2186c3" not found
Oct 31 02:12:01.040320 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:01.039841    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume podName:b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa nodeName:}" failed. No retries permitted until 2026-10-31 02:12:03.039822903 +0000 UTC m=+3080.988471296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume") pod "payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" (UID: "b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa") : configmap "payload-processing-plugins-e2e-extmod-a-2186c3" not found
Oct 31 02:12:03.114322 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:03.114275    5293 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-extmod-a-2186c3: configmap "payload-processing-plugins-e2e-extmod-a-2186c3" not found
Oct 31 02:12:03.114836 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:03.114382    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume podName:b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa nodeName:}" failed. No retries permitted until 2026-10-31 02:12:07.114360636 +0000 UTC m=+3085.063009024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume") pod "payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" (UID: "b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa") : configmap "payload-processing-plugins-e2e-extmod-a-2186c3" not found
Oct 31 02:12:04.079095 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.079020    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7" podUID="4e4f3ca0-7e72-417d-9750-3be06260bd17" containerName="payload-processing" containerID="cri-o://3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96" gracePeriod=25
Oct 31 02:12:04.395436 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.395412    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:12:04.426134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.426101    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"]
Oct 31 02:12:04.427406 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.427387    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0171f979-aedd-4d20-873c-c43ab02af336" containerName="revoke-keys"
Oct 31 02:12:04.427488 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.427409    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0171f979-aedd-4d20-873c-c43ab02af336" containerName="revoke-keys"
Oct 31 02:12:04.427488 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.427449    5293 cpu_manager.go:414] "containerMap: removing container" podUID="4e4f3ca0-7e72-417d-9750-3be06260bd17" containerName="payload-processing"
Oct 31 02:12:04.427488 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.427458    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4f3ca0-7e72-417d-9750-3be06260bd17" containerName="payload-processing"
Oct 31 02:12:04.427694 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.427671    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4f3ca0-7e72-417d-9750-3be06260bd17" containerName="payload-processing"
Oct 31 02:12:04.427746 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.427706    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0171f979-aedd-4d20-873c-c43ab02af336" containerName="revoke-keys"
Oct 31 02:12:04.432006 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.431987    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.435942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.435913    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"payload-processing-external-model-e2e--1ed73e85-dockercfg-mbxtd\""
Oct 31 02:12:04.439362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.439338    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"agc-cred-3b375805fa8e\""
Oct 31 02:12:04.440986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.439999    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"routing-overlay\""
Oct 31 02:12:04.440986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.440128    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ac7e0d\""
Oct 31 02:12:04.440986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.440189    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"openshift-service-ca.crt\""
Oct 31 02:12:04.440986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.440268    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"kube-root-ca.crt\""
Oct 31 02:12:04.443676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.443453    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"]
Oct 31 02:12:04.471916 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.471893    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4f3ca0-7e72-417d-9750-3be06260bd17-kube-api-access-m5x9r\" (UniqueName: \"kubernetes.io/projected/4e4f3ca0-7e72-417d-9750-3be06260bd17-kube-api-access-m5x9r\") pod \"4e4f3ca0-7e72-417d-9750-3be06260bd17\" (UID: \"4e4f3ca0-7e72-417d-9750-3be06260bd17\") "
Oct 31 02:12:04.472252 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.472232    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4f3ca0-7e72-417d-9750-3be06260bd17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f3ca0-7e72-417d-9750-3be06260bd17-plugins-config-volume\") pod \"4e4f3ca0-7e72-417d-9750-3be06260bd17\" (UID: \"4e4f3ca0-7e72-417d-9750-3be06260bd17\") "
Oct 31 02:12:04.472638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.472572    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.472638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.472623    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4f3ca0-7e72-417d-9750-3be06260bd17-plugins-config-volume" pod "4e4f3ca0-7e72-417d-9750-3be06260bd17" (UID: "4e4f3ca0-7e72-417d-9750-3be06260bd17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:04.472925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.472798    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.473124 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.473101    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjsjn\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-kube-api-access-xjsjn\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.473197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.473140    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.473375 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.473357    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4f3ca0-7e72-417d-9750-3be06260bd17-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:04.474451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.474428    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4f3ca0-7e72-417d-9750-3be06260bd17-kube-api-access-m5x9r" pod "4e4f3ca0-7e72-417d-9750-3be06260bd17" (UID: "4e4f3ca0-7e72-417d-9750-3be06260bd17"). InnerVolumeSpecName "kube-api-access-m5x9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:04.535958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.535934    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"agc-cred-142210d165ac\""
Oct 31 02:12:04.618616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.615776    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.618616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.616118    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjsjn\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-kube-api-access-xjsjn\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.618616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.616157    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.618616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.616362    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.618616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.616460    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5x9r\" (UniqueName: \"kubernetes.io/projected/4e4f3ca0-7e72-417d-9750-3be06260bd17-kube-api-access-m5x9r\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:04.618616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.617973    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.635069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.633153    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.653398 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.641772    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjsjn\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-kube-api-access-xjsjn\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.653398 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.650439    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.754542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.754509    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:04.896283 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.896242    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" podUID="b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa" containerName="payload-pre-processing" containerID="cri-o://c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663" gracePeriod=25
Oct 31 02:12:04.907943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.907916    5293 generic.go:356] "Generic (PLEG): container finished" podID="4e4f3ca0-7e72-417d-9750-3be06260bd17" containerID="3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96" exitCode=0
Oct 31 02:12:04.908103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.907972    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"
Oct 31 02:12:04.908103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.907997    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7" event={"ID":"4e4f3ca0-7e72-417d-9750-3be06260bd17","Type":"ContainerDied","Data":"3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96"}
Oct 31 02:12:04.908103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.908067    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7" event={"ID":"4e4f3ca0-7e72-417d-9750-3be06260bd17","Type":"ContainerDied","Data":"6a2e3c0d4d28936bb44b30cd041f783058cda123e3acd7b83924c9032b50182f"}
Oct 31 02:12:04.908103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.908095    5293 scope.go:122] "RemoveContainer" containerID="3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96"
Oct 31 02:12:04.974355 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.974325    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"]
Oct 31 02:12:04.982084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:04.982057    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2186c3-946d65949-m2fh7"]
Oct 31 02:12:05.012665 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.012642    5293 scope.go:122] "RemoveContainer" containerID="3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96"
Oct 31 02:12:05.013027 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:05.013003    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96\": container with ID starting with 3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96 not found: ID does not exist" containerID="3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96"
Oct 31 02:12:05.013155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.013081    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96"} err="failed to get container status \"3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96\": rpc error: code = NotFound desc = could not find container \"3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96\": container with ID starting with 3ff14ccee6e2b96690fb8966708ba1b08eadbd03e78c28b2eb1a40e357998f96 not found: ID does not exist"
Oct 31 02:12:05.053105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.053077    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"]
Oct 31 02:12:05.242765 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.242736    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:12:05.340889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.339315    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-kube-api-access-7w74r\" (UniqueName: \"kubernetes.io/projected/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-kube-api-access-7w74r\") pod \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\" (UID: \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\") "
Oct 31 02:12:05.340889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.339383    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume\") pod \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\" (UID: \"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa\") "
Oct 31 02:12:05.340889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.340772    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume" pod "b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa" (UID: "b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:05.352428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.352291    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-kube-api-access-7w74r" pod "b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa" (UID: "b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa"). InnerVolumeSpecName "kube-api-access-7w74r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:05.442461 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.442373    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w74r\" (UniqueName: \"kubernetes.io/projected/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-kube-api-access-7w74r\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:05.442461 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.442413    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:05.924012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.923974    5293 generic.go:356] "Generic (PLEG): container finished" podID="b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa" containerID="c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663" exitCode=0
Oct 31 02:12:05.924312 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.924094    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"
Oct 31 02:12:05.924312 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.924106    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" event={"ID":"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa","Type":"ContainerDied","Data":"c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663"}
Oct 31 02:12:05.924312 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.924219    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l" event={"ID":"b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa","Type":"ContainerDied","Data":"43ef3198c180bd25f10b6fe24a3835e7997634403367b8e7b5f1d0de68567cc6"}
Oct 31 02:12:05.924312 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.924272    5293 scope.go:122] "RemoveContainer" containerID="c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663"
Oct 31 02:12:05.926462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.926423    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69" event={"ID":"2092a552-680f-4bd0-9688-b7b0defe302d","Type":"ContainerStarted","Data":"d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c"}
Oct 31 02:12:05.926612 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.926482    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69" event={"ID":"2092a552-680f-4bd0-9688-b7b0defe302d","Type":"ContainerStarted","Data":"47088fd3e6fffc35596181f025d0c18f798fa73e1e48cdb58aa85697379f6498"}
Oct 31 02:12:05.926854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:05.926683    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:06.014064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:06.002811    5293 scope.go:122] "RemoveContainer" containerID="c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663"
Oct 31 02:12:06.036900 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:06.016194    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663\": container with ID starting with c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663 not found: ID does not exist" containerID="c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663"
Oct 31 02:12:06.036900 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:06.016243    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663"} err="failed to get container status \"c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663\": rpc error: code = NotFound desc = could not find container \"c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663\": container with ID starting with c00d0df95282fa12ea9761a9bad951aff5371216e08e10df8e173c71f9fd6663 not found: ID does not exist"
Oct 31 02:12:06.066063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:06.056791    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69" podStartSLOduration=25.056766399 podStartE2EDuration="25.056766399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:06.026206085 +0000 UTC m=+3083.974854478" watchObservedRunningTime="2026-10-31 02:12:06.056766399 +0000 UTC m=+3084.005414792"
Oct 31 02:12:06.066063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:06.061512    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"]
Oct 31 02:12:06.073067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:06.069571    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2186c3-675fbc4db7-f654l"]
Oct 31 02:12:06.182985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:06.182904    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4f3ca0-7e72-417d-9750-3be06260bd17" path="/var/lib/kubelet/pods/4e4f3ca0-7e72-417d-9750-3be06260bd17/volumes"
Oct 31 02:12:06.183695 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:06.183673    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa" path="/var/lib/kubelet/pods/b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa/volumes"
Oct 31 02:12:11.781254 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:11.781215    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:12:12.556064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:12.555109    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"]
Oct 31 02:12:12.591064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:12.588967    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2186c3-7gtzc"]
Oct 31 02:12:14.177410 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:14.177374    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0171f979-aedd-4d20-873c-c43ab02af336" path="/var/lib/kubelet/pods/0171f979-aedd-4d20-873c-c43ab02af336/volumes"
Oct 31 02:12:16.945544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:16.945512    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:12:18.319506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:18.319463    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:12:19.895217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:19.895159    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"]
Oct 31 02:12:19.896463 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:19.896422    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy" containerID="cri-o://d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb" gracePeriod=30
Oct 31 02:12:20.570031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.569371    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"]
Oct 31 02:12:20.570031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.569443    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"]
Oct 31 02:12:20.570031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.569641    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy" containerID="cri-o://fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4" gracePeriod=30
Oct 31 02:12:20.585979 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.585540    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae625aeb-l2899"]
Oct 31 02:12:20.603095 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.602385    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"]
Oct 31 02:12:20.604817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.603874    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa" containerName="payload-pre-processing"
Oct 31 02:12:20.604817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.603899    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa" containerName="payload-pre-processing"
Oct 31 02:12:20.604817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.604179    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53a20b4-3a7c-4d66-8d59-3bd2ac8965aa" containerName="payload-pre-processing"
Oct 31 02:12:20.624172 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.623568    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"]
Oct 31 02:12:20.624172 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.623690    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.632211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.631602    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:12:20.632211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.631670    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:12:20.710062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.704848    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.710062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.704927    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlxk\" (UniqueName: \"kubernetes.io/projected/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-kube-api-access-njlxk\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.710062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.705087    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.834519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.834415    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.834762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.834519    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlxk\" (UniqueName: \"kubernetes.io/projected/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-kube-api-access-njlxk\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.834762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.834682    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.846222 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.846179    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.848104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.848064    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.854927 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.854896    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlxk\" (UniqueName: \"kubernetes.io/projected/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-kube-api-access-njlxk\") pod \"maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:20.981636 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:20.981597    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:21.383739 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:21.383660    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"]
Oct 31 02:12:21.386900 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:12:21.386850    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fef4b8_fba8_44ac_be5e_3a12ed2bb7d2.slice/crio-6f2544126b18ed0387f8c4d9cbcd30a62bbb1b1f16d175d99194cd4f95ea73cd WatchSource:0}: Error finding container 6f2544126b18ed0387f8c4d9cbcd30a62bbb1b1f16d175d99194cd4f95ea73cd: Status 404 returned error can't find the container with id 6f2544126b18ed0387f8c4d9cbcd30a62bbb1b1f16d175d99194cd4f95ea73cd
Oct 31 02:12:21.705062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:21.704831    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q" event={"ID":"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2","Type":"ContainerStarted","Data":"6f2544126b18ed0387f8c4d9cbcd30a62bbb1b1f16d175d99194cd4f95ea73cd"}
Oct 31 02:12:22.220007 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:22.219795    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3175419f-620d-4893-b501-8f146f42a5f0" path="/var/lib/kubelet/pods/3175419f-620d-4893-b501-8f146f42a5f0/volumes"
Oct 31 02:12:22.798065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:22.797197    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:22.933997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:22.923213    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:22.933997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:22.933125    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:22.985068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:22.978015    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:22.985640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:22.984267    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q" event={"ID":"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2","Type":"ContainerStarted","Data":"039e900f8f3fbdf50f9b6ab4a608f3048cfba22d7027c07cd6380f2db85ce998"}
Oct 31 02:12:23.008638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.006142    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 02:12:23.138124 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.126249    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:23.254070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.187853    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:23.356510 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.187899    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:12:23.383068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.188994    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:23.383068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.208012    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:23.383068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.283331    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:23.809063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.807989    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:23.926926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.926879    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:12:23.957065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.929509    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:12:23.966086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.941159    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:23.966958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.966930    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:12:23.990206 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.990164    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Oct 31 02:12:23.991198 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:23.991172    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:12:24.014627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.007202    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:24.015575 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.015552    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:12:24.272846 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.249174    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:12:24.272846 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.256904    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:24.272846 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.257838    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:12:24.357166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.357119    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 31 02:12:24.458658 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.447735    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:12:24.458658 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.447802    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:12:24.458658 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.450220    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:12:24.458658 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.454254    5293 generic.go:356] "Generic (PLEG): container finished" podID="f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2" containerID="039e900f8f3fbdf50f9b6ab4a608f3048cfba22d7027c07cd6380f2db85ce998" exitCode=0
Oct 31 02:12:24.458658 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.454560    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q" event={"ID":"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2","Type":"ContainerDied","Data":"039e900f8f3fbdf50f9b6ab4a608f3048cfba22d7027c07cd6380f2db85ce998"}
Oct 31 02:12:24.562112 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:24.550638    5293 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6e1bafd9570a: secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:24.562112 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:24.556094    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8: secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:24.562112 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:24.556205    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials podName:41b073a6-5030-4f71-b024-b2d0c436051b nodeName:}" failed. No retries permitted until 2026-10-31 02:12:25.056180343 +0000 UTC m=+3103.004828739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials") pod "payload-processing-external-model-57745f7bf-vqzk8" (UID: "41b073a6-5030-4f71-b024-b2d0c436051b") : secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:24.680896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.679652    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:12:24.726401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.726295    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:12:24.776500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.776443    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw" podUID="808265a9-b1bb-4c13-ba57-6d4aa0e91931" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded"
Oct 31 02:12:24.844664 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.844550    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:24.858267 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.858236    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:12:24.874694 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.874668    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw"
Oct 31 02:12:24.886091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.879446    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:12:24.944733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:24.944698    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:12:25.106545 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:25.106142    5293 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6e1bafd9570a: secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:25.106545 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:25.106169    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8: secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:25.106545 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:25.106225    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials podName:41b073a6-5030-4f71-b024-b2d0c436051b nodeName:}" failed. No retries permitted until 2026-10-31 02:12:26.106208862 +0000 UTC m=+3104.054857249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials") pod "payload-processing-external-model-57745f7bf-vqzk8" (UID: "41b073a6-5030-4f71-b024-b2d0c436051b") : secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:25.192980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.192942    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:25.206984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.206954    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-xks-ca\") pod \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") "
Oct 31 02:12:25.207223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.207207    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-kube-api-access-njlxk\" (UniqueName: \"kubernetes.io/projected/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-kube-api-access-njlxk\") pod \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") "
Oct 31 02:12:25.207306 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.207259    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-openshift-ca\") pod \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\" (UID: \"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2\") "
Oct 31 02:12:25.207670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.207642    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-openshift-ca" pod "f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2" (UID: "f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:25.208241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.208219    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.210281 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.210255    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-kube-api-access-njlxk" pod "f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2" (UID: "f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2"). InnerVolumeSpecName "kube-api-access-njlxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:25.210393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.210354    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-xks-ca" pod "f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2" (UID: "f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:12:25.310980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.310936    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njlxk\" (UniqueName: \"kubernetes.io/projected/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-kube-api-access-njlxk\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.310980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.310976    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.362720 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.362630    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:12:25.462577 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.462546    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"]
Oct 31 02:12:25.475850 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.475824    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"
Oct 31 02:12:25.476059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.475866    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q" event={"ID":"f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2","Type":"ContainerDied","Data":"6f2544126b18ed0387f8c4d9cbcd30a62bbb1b1f16d175d99194cd4f95ea73cd"}
Oct 31 02:12:25.476059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.475910    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f2544126b18ed0387f8c4d9cbcd30a62bbb1b1f16d175d99194cd4f95ea73cd"
Oct 31 02:12:25.477709 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.477680    5293 generic.go:356] "Generic (PLEG): container finished" podID="57b0a3e1-d848-4931-8bec-bea42f843751" containerID="d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb" exitCode=0
Oct 31 02:12:25.477844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.477782    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"
Oct 31 02:12:25.477844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.477809    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" event={"ID":"57b0a3e1-d848-4931-8bec-bea42f843751","Type":"ContainerDied","Data":"d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb"}
Oct 31 02:12:25.477971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.477870    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr" event={"ID":"57b0a3e1-d848-4931-8bec-bea42f843751","Type":"ContainerDied","Data":"99df0fa9af20a975d55ef51bcd0f476c13c122afc9d4a18b24df584dae7c8b49"}
Oct 31 02:12:25.477971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.477902    5293 scope.go:122] "RemoveContainer" containerID="d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb"
Oct 31 02:12:25.493188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.493172    5293 scope.go:122] "RemoveContainer" containerID="d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb"
Oct 31 02:12:25.493483 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:25.493463    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb\": container with ID starting with d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb not found: ID does not exist" containerID="d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb"
Oct 31 02:12:25.493551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.493491    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb"} err="failed to get container status \"d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb\": rpc error: code = NotFound desc = could not find container \"d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb\": container with ID starting with d3eca28bf077f10cb61d6e53fb0508a0ffb1bab9887eddae89d867dde80eeedb not found: ID does not exist"
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.560292    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-istio-token\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-istio-token\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.560355    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-kube-api-access-s77lv\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-kube-api-access-s77lv\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.560415    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-socket\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.560479    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-credential-socket\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.560509    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-certs\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.560635    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b0a3e1-d848-4931-8bec-bea42f843751-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0a3e1-d848-4931-8bec-bea42f843751-istiod-ca-cert\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.560733    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57b0a3e1-d848-4931-8bec-bea42f843751-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0a3e1-d848-4931-8bec-bea42f843751-istio-podinfo\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.560975    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-envoy\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.568066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.561077    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-data\") pod \"57b0a3e1-d848-4931-8bec-bea42f843751\" (UID: \"57b0a3e1-d848-4931-8bec-bea42f843751\") "
Oct 31 02:12:25.576672 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.568669    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-socket" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.576672 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.568884    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-credential-socket" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.576672 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.569289    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-certs" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.576672 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.569851    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b0a3e1-d848-4931-8bec-bea42f843751-istiod-ca-cert" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:25.576672 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.576127    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-data" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.599136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.590167    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-envoy" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.599136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.590259    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57b0a3e1-d848-4931-8bec-bea42f843751-istio-podinfo" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:12:25.617133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.612229    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-istio-token" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:25.642099 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.637462    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-kube-api-access-s77lv" pod "57b0a3e1-d848-4931-8bec-bea42f843751" (UID: "57b0a3e1-d848-4931-8bec-bea42f843751"). InnerVolumeSpecName "kube-api-access-s77lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:25.664806 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664763    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.664806 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664796    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s77lv\" (UniqueName: \"kubernetes.io/projected/57b0a3e1-d848-4931-8bec-bea42f843751-kube-api-access-s77lv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.664806 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664812    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.665203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664825    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.665203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664840    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.665203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664853    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b0a3e1-d848-4931-8bec-bea42f843751-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.665203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664865    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b0a3e1-d848-4931-8bec-bea42f843751-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.665203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664878    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.665203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.664890    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b0a3e1-d848-4931-8bec-bea42f843751-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.825232 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.825146    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"]
Oct 31 02:12:25.833322 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.833291    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-vhnbr"]
Oct 31 02:12:25.871778 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.871707    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:12:25.969966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.969919    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-certs\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970023    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-data\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970182    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-envoy\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970283 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970229    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-socket\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970283 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970244    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-certs" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.970283 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970257    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-data" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.970454 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970305    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istiod-ca-cert\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970454 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970404    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-podinfo\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970466    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-token\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-token\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970542    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-credential-socket\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970595    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-socket" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.970674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970648    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-kube-api-access-2tdvx\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-kube-api-access-2tdvx\") pod \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\" (UID: \"d367975b-ff2a-4a3a-9cd3-a5cd32820603\") "
Oct 31 02:12:25.970785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970692    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istiod-ca-cert" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:25.970898 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.970875    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-credential-socket" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.971602 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.971579    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.971724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.971605    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.971724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.971621    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.971724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.971636    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.971724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.971649    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.972834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.972807    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-envoy" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.972989 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.972971    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-podinfo" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:12:25.973484 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.973467    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-kube-api-access-2tdvx" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "kube-api-access-2tdvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:25.973545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:25.973493    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-token" pod "d367975b-ff2a-4a3a-9cd3-a5cd32820603" (UID: "d367975b-ff2a-4a3a-9cd3-a5cd32820603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:26.074133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.074103    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:26.074275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.074136    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:26.074275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.074150    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tdvx\" (UniqueName: \"kubernetes.io/projected/d367975b-ff2a-4a3a-9cd3-a5cd32820603-kube-api-access-2tdvx\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:26.074275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.074166    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d367975b-ff2a-4a3a-9cd3-a5cd32820603-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:26.181424 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:26.181392    5293 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6e1bafd9570a: secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:26.181668 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:26.181654    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8: secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:26.181832 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:26.181819    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials podName:41b073a6-5030-4f71-b024-b2d0c436051b nodeName:}" failed. No retries permitted until 2026-10-31 02:12:28.181798498 +0000 UTC m=+3106.130446882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials") pod "payload-processing-external-model-57745f7bf-vqzk8" (UID: "41b073a6-5030-4f71-b024-b2d0c436051b") : secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:26.190739 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.190703    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" path="/var/lib/kubelet/pods/57b0a3e1-d848-4931-8bec-bea42f843751/volumes"
Oct 31 02:12:26.489495 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.489403    5293 generic.go:356] "Generic (PLEG): container finished" podID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerID="fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4" exitCode=0
Oct 31 02:12:26.489495 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.489476    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"
Oct 31 02:12:26.489495 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.489481    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" event={"ID":"d367975b-ff2a-4a3a-9cd3-a5cd32820603","Type":"ContainerDied","Data":"fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4"}
Oct 31 02:12:26.490117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.489542    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9" event={"ID":"d367975b-ff2a-4a3a-9cd3-a5cd32820603","Type":"ContainerDied","Data":"23971f19af819e2c97443e4420f40613522f7c2a9936a157ca76f3e822c58178"}
Oct 31 02:12:26.490117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.489568    5293 scope.go:122] "RemoveContainer" containerID="fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4"
Oct 31 02:12:26.502470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.502451    5293 scope.go:122] "RemoveContainer" containerID="fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4"
Oct 31 02:12:26.502755 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:26.502731    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4\": container with ID starting with fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4 not found: ID does not exist" containerID="fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4"
Oct 31 02:12:26.502863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.502762    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4"} err="failed to get container status \"fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4\": rpc error: code = NotFound desc = could not find container \"fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4\": container with ID starting with fbecc2b23113e7595f135747342bbb819c6a508cf6d76dd315ad89558f920ac4 not found: ID does not exist"
Oct 31 02:12:26.512533 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.512508    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"]
Oct 31 02:12:26.518348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:26.518325    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-sz5h9"]
Oct 31 02:12:28.028625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:28.028586    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"]
Oct 31 02:12:28.033200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:28.033174    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac4f642-9b4mj"]
Oct 31 02:12:28.176467 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:28.176433    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" path="/var/lib/kubelet/pods/d367975b-ff2a-4a3a-9cd3-a5cd32820603/volumes"
Oct 31 02:12:28.176977 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:28.176962    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de172b2b-f10a-48dc-bed8-4b3ff0788ffb" path="/var/lib/kubelet/pods/de172b2b-f10a-48dc-bed8-4b3ff0788ffb/volumes"
Oct 31 02:12:28.247708 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:28.247669    5293 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6e1bafd9570a: secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:28.247708 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:28.247698    5293 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8: secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:28.247976 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:28.247751    5293 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials podName:41b073a6-5030-4f71-b024-b2d0c436051b nodeName:}" failed. No retries permitted until 2026-10-31 02:12:32.247736052 +0000 UTC m=+3110.196384422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials") pod "payload-processing-external-model-57745f7bf-vqzk8" (UID: "41b073a6-5030-4f71-b024-b2d0c436051b") : secret "agc-cred-6e1bafd9570a" not found
Oct 31 02:12:30.482083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:30.481143    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8" podUID="41b073a6-5030-4f71-b024-b2d0c436051b" containerName="payload-processing" containerID="cri-o://6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514" gracePeriod=25
Oct 31 02:12:30.662546 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:30.662512    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"]
Oct 31 02:12:30.676297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:30.676224    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"]
Oct 31 02:12:30.916322 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:30.916295    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:12:31.060068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.055782    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-kube-api-access-tf6j2\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-kube-api-access-tf6j2\") pod \"41b073a6-5030-4f71-b024-b2d0c436051b\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") "
Oct 31 02:12:31.060068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.056067    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials\") pod \"41b073a6-5030-4f71-b024-b2d0c436051b\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") "
Oct 31 02:12:31.060068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.056268    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-plugins-config-volume\") pod \"41b073a6-5030-4f71-b024-b2d0c436051b\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") "
Oct 31 02:12:31.060068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.056322    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-routing-overlay-volume\") pod \"41b073a6-5030-4f71-b024-b2d0c436051b\" (UID: \"41b073a6-5030-4f71-b024-b2d0c436051b\") "
Oct 31 02:12:31.065061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.061667    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-routing-overlay-volume" pod "41b073a6-5030-4f71-b024-b2d0c436051b" (UID: "41b073a6-5030-4f71-b024-b2d0c436051b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:31.065061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.062187    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-plugins-config-volume" pod "41b073a6-5030-4f71-b024-b2d0c436051b" (UID: "41b073a6-5030-4f71-b024-b2d0c436051b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:31.097145 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.083289    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials" pod "41b073a6-5030-4f71-b024-b2d0c436051b" (UID: "41b073a6-5030-4f71-b024-b2d0c436051b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:31.141329 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.138728    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-kube-api-access-tf6j2" pod "41b073a6-5030-4f71-b024-b2d0c436051b" (UID: "41b073a6-5030-4f71-b024-b2d0c436051b"). InnerVolumeSpecName "kube-api-access-tf6j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:31.165161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.159948    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6j2\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-kube-api-access-tf6j2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:31.165161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.159989    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41b073a6-5030-4f71-b024-b2d0c436051b-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:31.165161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.160008    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:31.165161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.160056    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41b073a6-5030-4f71-b024-b2d0c436051b-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:31.646017 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.634158    5293 generic.go:356] "Generic (PLEG): container finished" podID="41b073a6-5030-4f71-b024-b2d0c436051b" containerID="6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514" exitCode=0
Oct 31 02:12:31.646017 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.634369    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8" event={"ID":"41b073a6-5030-4f71-b024-b2d0c436051b","Type":"ContainerDied","Data":"6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514"}
Oct 31 02:12:31.646017 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.634428    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8" event={"ID":"41b073a6-5030-4f71-b024-b2d0c436051b","Type":"ContainerDied","Data":"f64377c749bb717f18035c09b86c259270882ad81e11a1219521d3c68c6fb540"}
Oct 31 02:12:31.646017 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.634459    5293 scope.go:122] "RemoveContainer" containerID="6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514"
Oct 31 02:12:31.646017 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.636249    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"
Oct 31 02:12:31.659080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.659054    5293 scope.go:122] "RemoveContainer" containerID="6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514"
Oct 31 02:12:31.659458 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:31.659429    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514\": container with ID starting with 6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514 not found: ID does not exist" containerID="6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514"
Oct 31 02:12:31.659538 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.659472    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514"} err="failed to get container status \"6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514\": rpc error: code = NotFound desc = could not find container \"6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514\": container with ID starting with 6282a57d688371343a82f5cd47d5ed522f079a0854998be076e9532cb3039514 not found: ID does not exist"
Oct 31 02:12:31.671367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.671337    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"]
Oct 31 02:12:31.676696 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:31.676675    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57745f7bf-vqzk8"]
Oct 31 02:12:32.176656 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:32.176596    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b073a6-5030-4f71-b024-b2d0c436051b" path="/var/lib/kubelet/pods/41b073a6-5030-4f71-b024-b2d0c436051b/volumes"
Oct 31 02:12:35.673940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:35.668751    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh" podUID="3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" containerName="payload-processing" containerID="cri-o://c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268" gracePeriod=25
Oct 31 02:12:35.687238 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:35.687198    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw" podUID="872a4172-8fca-4b53-8bb4-aa0ae3010b58" containerName="payload-pre-processing" containerID="cri-o://fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73" gracePeriod=25
Oct 31 02:12:39.037840 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:39.037802    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"]
Oct 31 02:12:39.046211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:39.046179    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9224db9-7xzhp"]
Oct 31 02:12:40.178291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:40.178250    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0515c49f-34cd-4eff-b7f0-d42e4070d93a" path="/var/lib/kubelet/pods/0515c49f-34cd-4eff-b7f0-d42e4070d93a/volumes"
Oct 31 02:12:40.335064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:40.328033    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw" podUID="872a4172-8fca-4b53-8bb4-aa0ae3010b58" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:12:41.288275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:41.288237    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh" podUID="3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:12:41.612794 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:41.612718    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"]
Oct 31 02:12:41.617818 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:41.617785    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-df9d42-5mp8q"]
Oct 31 02:12:42.061100 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:42.061061    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:12:42.242899 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:42.242857    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2" path="/var/lib/kubelet/pods/f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2/volumes"
Oct 31 02:12:47.025937 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:47.025897    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"]
Oct 31 02:12:47.030296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:47.030264    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7100c22f-pklht"]
Oct 31 02:12:48.239585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:48.239510    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8c2477-c524-4630-9ac9-0b5edacad877" path="/var/lib/kubelet/pods/7b8c2477-c524-4630-9ac9-0b5edacad877/volumes"
Oct 31 02:12:49.161276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.161243    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"]
Oct 31 02:12:49.161940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.161906    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" containerID="cri-o://c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25" gracePeriod=30
Oct 31 02:12:49.412698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.412619    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:12:49.413501 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.413240    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:12:49.731671 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.731584    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"]
Oct 31 02:12:49.732505 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732486    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy"
Oct 31 02:12:49.732505 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732505    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy"
Oct 31 02:12:49.732635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732525    5293 cpu_manager.go:414] "containerMap: removing container" podUID="41b073a6-5030-4f71-b024-b2d0c436051b" containerName="payload-processing"
Oct 31 02:12:49.732635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732534    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b073a6-5030-4f71-b024-b2d0c436051b" containerName="payload-processing"
Oct 31 02:12:49.732635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732548    5293 cpu_manager.go:414] "containerMap: removing container" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy"
Oct 31 02:12:49.732635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732553    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy"
Oct 31 02:12:49.732635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732588    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2" containerName="revoke-keys"
Oct 31 02:12:49.732635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732593    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2" containerName="revoke-keys"
Oct 31 02:12:49.732829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732710    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d367975b-ff2a-4a3a-9cd3-a5cd32820603" containerName="istio-proxy"
Oct 31 02:12:49.732829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732720    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b073a6-5030-4f71-b024-b2d0c436051b" containerName="payload-processing"
Oct 31 02:12:49.732829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732728    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b0a3e1-d848-4931-8bec-bea42f843751" containerName="istio-proxy"
Oct 31 02:12:49.732829 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.732741    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3fef4b8-fba8-44ac-be5e-3a12ed2bb7d2" containerName="revoke-keys"
Oct 31 02:12:49.738151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.738130    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.742836 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.742809    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:12:49.743508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.743482    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:12:49.769020 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.768989    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"]
Oct 31 02:12:49.788567 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.788536    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4441c772-db48-4d8a-8734-73483467fe02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.788751 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.788673    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2ml\" (UniqueName: \"kubernetes.io/projected/4441c772-db48-4d8a-8734-73483467fe02-kube-api-access-jn2ml\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.788873 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.788858    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4441c772-db48-4d8a-8734-73483467fe02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.930903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.930839    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4441c772-db48-4d8a-8734-73483467fe02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.931174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.930942    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2ml\" (UniqueName: \"kubernetes.io/projected/4441c772-db48-4d8a-8734-73483467fe02-kube-api-access-jn2ml\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.931174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.931111    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4441c772-db48-4d8a-8734-73483467fe02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.932194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.932163    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4441c772-db48-4d8a-8734-73483467fe02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.970857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.967479    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2ml\" (UniqueName: \"kubernetes.io/projected/4441c772-db48-4d8a-8734-73483467fe02-kube-api-access-jn2ml\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:49.970857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:49.970528    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4441c772-db48-4d8a-8734-73483467fe02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:50.070170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:50.070016    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:50.281059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:50.281011    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw" podUID="872a4172-8fca-4b53-8bb4-aa0ae3010b58" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:12:50.360288 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:12:50.360257    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4441c772_db48_4d8a_8734_73483467fe02.slice/crio-93bc93d17ac4b1c670f73b819bab33f060ffdb53f154910b397a00003824da9c WatchSource:0}: Error finding container 93bc93d17ac4b1c670f73b819bab33f060ffdb53f154910b397a00003824da9c: Status 404 returned error can't find the container with id 93bc93d17ac4b1c670f73b819bab33f060ffdb53f154910b397a00003824da9c
Oct 31 02:12:50.360569 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:50.360545    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"]
Oct 31 02:12:51.037278 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:51.037242    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z" event={"ID":"4441c772-db48-4d8a-8734-73483467fe02","Type":"ContainerStarted","Data":"ddf7aaddeb829506dd7c73efd1b88486bf544ea391dd9ba91f4c464594d7dbba"}
Oct 31 02:12:51.037993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:51.037310    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z" event={"ID":"4441c772-db48-4d8a-8734-73483467fe02","Type":"ContainerStarted","Data":"93bc93d17ac4b1c670f73b819bab33f060ffdb53f154910b397a00003824da9c"}
Oct 31 02:12:51.057436 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:51.057392    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z" podStartSLOduration=2.05737819 podStartE2EDuration="2.05737819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:51.053856294 +0000 UTC m=+3129.002504686" watchObservedRunningTime="2026-10-31 02:12:51.05737819 +0000 UTC m=+3129.006026615"
Oct 31 02:12:51.344162 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:51.339711    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh" podUID="3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:12:52.059817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:52.057437    5293 generic.go:356] "Generic (PLEG): container finished" podID="4441c772-db48-4d8a-8734-73483467fe02" containerID="ddf7aaddeb829506dd7c73efd1b88486bf544ea391dd9ba91f4c464594d7dbba" exitCode=0
Oct 31 02:12:52.059817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:52.057545    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z" event={"ID":"4441c772-db48-4d8a-8734-73483467fe02","Type":"ContainerDied","Data":"ddf7aaddeb829506dd7c73efd1b88486bf544ea391dd9ba91f4c464594d7dbba"}
Oct 31 02:12:53.283495 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.283470    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:53.371122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.365023    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4441c772-db48-4d8a-8734-73483467fe02-kube-api-access-jn2ml\" (UniqueName: \"kubernetes.io/projected/4441c772-db48-4d8a-8734-73483467fe02-kube-api-access-jn2ml\") pod \"4441c772-db48-4d8a-8734-73483467fe02\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") "
Oct 31 02:12:53.371122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.365297    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4441c772-db48-4d8a-8734-73483467fe02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4441c772-db48-4d8a-8734-73483467fe02-maas-api-openshift-ca\") pod \"4441c772-db48-4d8a-8734-73483467fe02\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") "
Oct 31 02:12:53.371122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.365465    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4441c772-db48-4d8a-8734-73483467fe02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4441c772-db48-4d8a-8734-73483467fe02-maas-api-xks-ca\") pod \"4441c772-db48-4d8a-8734-73483467fe02\" (UID: \"4441c772-db48-4d8a-8734-73483467fe02\") "
Oct 31 02:12:53.433879 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.431517    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4441c772-db48-4d8a-8734-73483467fe02-maas-api-openshift-ca" pod "4441c772-db48-4d8a-8734-73483467fe02" (UID: "4441c772-db48-4d8a-8734-73483467fe02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:53.434569 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.434531    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4441c772-db48-4d8a-8734-73483467fe02-maas-api-xks-ca" pod "4441c772-db48-4d8a-8734-73483467fe02" (UID: "4441c772-db48-4d8a-8734-73483467fe02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:12:53.480518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.480486    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4441c772-db48-4d8a-8734-73483467fe02-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:53.480889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.480869    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4441c772-db48-4d8a-8734-73483467fe02-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:53.481817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.481788    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4441c772-db48-4d8a-8734-73483467fe02-kube-api-access-jn2ml" pod "4441c772-db48-4d8a-8734-73483467fe02" (UID: "4441c772-db48-4d8a-8734-73483467fe02"). InnerVolumeSpecName "kube-api-access-jn2ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:53.583299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:53.583261    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2ml\" (UniqueName: \"kubernetes.io/projected/4441c772-db48-4d8a-8734-73483467fe02-kube-api-access-jn2ml\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.172362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.172325    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"
Oct 31 02:12:54.187963 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.187929    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z" event={"ID":"4441c772-db48-4d8a-8734-73483467fe02","Type":"ContainerDied","Data":"93bc93d17ac4b1c670f73b819bab33f060ffdb53f154910b397a00003824da9c"}
Oct 31 02:12:54.188104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.188001    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93bc93d17ac4b1c670f73b819bab33f060ffdb53f154910b397a00003824da9c"
Oct 31 02:12:54.477769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.477741    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:12:54.611632 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.611604    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-envoy\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.611830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.611662    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-credential-socket\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.611899 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.611827    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-certs\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.611974 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.611905    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-socket\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.612085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.611970    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istiod-ca-cert\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.612085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612022    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-credential-socket" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:54.612223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612089    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-kube-api-access-mt2m5\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-kube-api-access-mt2m5\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.612223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612109    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-certs" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:54.612223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612135    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-socket" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:54.612223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612160    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-data\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.612492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612227    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-podinfo\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.612492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612322    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-token\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-token\") pod \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\" (UID: \"fc5aa93f-da9b-41e2-acc1-155a54a3767d\") "
Oct 31 02:12:54.612492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612479    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istiod-ca-cert" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:54.612670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.612550    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-data" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:54.613470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.613442    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.613590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.613473    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.613590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.613488    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.613590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.613503    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.613590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.613517    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.614352 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.614306    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-envoy" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:54.614706 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.614688    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-kube-api-access-mt2m5" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "kube-api-access-mt2m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:54.615297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.615272    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-podinfo" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:12:54.616444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.616421    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-token" pod "fc5aa93f-da9b-41e2-acc1-155a54a3767d" (UID: "fc5aa93f-da9b-41e2-acc1-155a54a3767d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:54.731149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.730913    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt2m5\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-kube-api-access-mt2m5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.731149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.730964    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.731149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.730980    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:54.731149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:54.730994    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5aa93f-da9b-41e2-acc1-155a54a3767d-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:55.182857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.182827    5293 generic.go:356] "Generic (PLEG): container finished" podID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerID="c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25" exitCode=0
Oct 31 02:12:55.183078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.182879    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"
Oct 31 02:12:55.183078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.182919    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" event={"ID":"fc5aa93f-da9b-41e2-acc1-155a54a3767d","Type":"ContainerDied","Data":"c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25"}
Oct 31 02:12:55.183078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.182987    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng" event={"ID":"fc5aa93f-da9b-41e2-acc1-155a54a3767d","Type":"ContainerDied","Data":"5f8e90837a6c4847dfdbf0e48762c4758a45cd61a611a3c51f41e6b677dfd265"}
Oct 31 02:12:55.183078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.183017    5293 scope.go:122] "RemoveContainer" containerID="c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25"
Oct 31 02:12:55.196522 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.196487    5293 scope.go:122] "RemoveContainer" containerID="b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519"
Oct 31 02:12:55.209991 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.209932    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"]
Oct 31 02:12:55.211307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.211283    5293 scope.go:122] "RemoveContainer" containerID="c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25"
Oct 31 02:12:55.211674 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:55.211636    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25\": container with ID starting with c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25 not found: ID does not exist" containerID="c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25"
Oct 31 02:12:55.211796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.211694    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25"} err="failed to get container status \"c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25\": rpc error: code = NotFound desc = could not find container \"c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25\": container with ID starting with c6135f2303a64c08e5b7593881bb47b160bd96c71e7601c8a9028be1f8d21e25 not found: ID does not exist"
Oct 31 02:12:55.211796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.211719    5293 scope.go:122] "RemoveContainer" containerID="b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519"
Oct 31 02:12:55.212129 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:55.212104    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519\": container with ID starting with b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519 not found: ID does not exist" containerID="b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519"
Oct 31 02:12:55.212217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.212137    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519"} err="failed to get container status \"b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519\": rpc error: code = NotFound desc = could not find container \"b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519\": container with ID starting with b35b65d2fbee515772d63cf8c099d42835f12e074bf3cfed5d0a038fce77a519 not found: ID does not exist"
Oct 31 02:12:55.215133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:55.215058    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-df9d42-openshift-default-5df79d78fc-l2dng"]
Oct 31 02:12:56.020149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.020122    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:12:56.076582 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.076554    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:12:56.167415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.164087    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-kube-api-access-2dvpd\" (UniqueName: \"kubernetes.io/projected/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-kube-api-access-2dvpd\") pod \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\" (UID: \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\") "
Oct 31 02:12:56.167415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.164190    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-plugins-config-volume\") pod \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\" (UID: \"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b\") "
Oct 31 02:12:56.167415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.164297    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872a4172-8fca-4b53-8bb4-aa0ae3010b58-kube-api-access-t8lfg\" (UniqueName: \"kubernetes.io/projected/872a4172-8fca-4b53-8bb4-aa0ae3010b58-kube-api-access-t8lfg\") pod \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\" (UID: \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\") "
Oct 31 02:12:56.167415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.164517    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872a4172-8fca-4b53-8bb4-aa0ae3010b58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872a4172-8fca-4b53-8bb4-aa0ae3010b58-plugins-config-volume\") pod \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\" (UID: \"872a4172-8fca-4b53-8bb4-aa0ae3010b58\") "
Oct 31 02:12:56.177348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.177301    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872a4172-8fca-4b53-8bb4-aa0ae3010b58-plugins-config-volume" pod "872a4172-8fca-4b53-8bb4-aa0ae3010b58" (UID: "872a4172-8fca-4b53-8bb4-aa0ae3010b58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:56.258149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.257361    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-plugins-config-volume" pod "3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" (UID: "3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:56.273031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.272992    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872a4172-8fca-4b53-8bb4-aa0ae3010b58-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:56.273258 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.273239    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:56.282257 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.282225    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-kube-api-access-2dvpd" pod "3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" (UID: "3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b"). InnerVolumeSpecName "kube-api-access-2dvpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:56.282415 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.282272    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872a4172-8fca-4b53-8bb4-aa0ae3010b58-kube-api-access-t8lfg" pod "872a4172-8fca-4b53-8bb4-aa0ae3010b58" (UID: "872a4172-8fca-4b53-8bb4-aa0ae3010b58"). InnerVolumeSpecName "kube-api-access-t8lfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:56.296939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.296906    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" path="/var/lib/kubelet/pods/fc5aa93f-da9b-41e2-acc1-155a54a3767d/volumes"
Oct 31 02:12:56.313901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.313860    5293 generic.go:356] "Generic (PLEG): container finished" podID="872a4172-8fca-4b53-8bb4-aa0ae3010b58" containerID="fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73" exitCode=0
Oct 31 02:12:56.314158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.314134    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw" event={"ID":"872a4172-8fca-4b53-8bb4-aa0ae3010b58","Type":"ContainerDied","Data":"fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73"}
Oct 31 02:12:56.314265 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.314161    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"
Oct 31 02:12:56.314265 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.314196    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw" event={"ID":"872a4172-8fca-4b53-8bb4-aa0ae3010b58","Type":"ContainerDied","Data":"5ef2780701c293584e1212ab5ac4293d487263063f1da0911feede72ea5d8b43"}
Oct 31 02:12:56.314265 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.314224    5293 scope.go:122] "RemoveContainer" containerID="fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73"
Oct 31 02:12:56.316960 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.316935    5293 generic.go:356] "Generic (PLEG): container finished" podID="3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" containerID="c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268" exitCode=0
Oct 31 02:12:56.317200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.317021    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"
Oct 31 02:12:56.317368 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.317080    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh" event={"ID":"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b","Type":"ContainerDied","Data":"c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268"}
Oct 31 02:12:56.317500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.317483    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh" event={"ID":"3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b","Type":"ContainerDied","Data":"e48c48423168b8e6a788faa1273600695bf863e5de0aef99c0b9c25a66bf9885"}
Oct 31 02:12:56.337560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.337533    5293 scope.go:122] "RemoveContainer" containerID="fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73"
Oct 31 02:12:56.337893 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:56.337854    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73\": container with ID starting with fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73 not found: ID does not exist" containerID="fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73"
Oct 31 02:12:56.337985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.337905    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73"} err="failed to get container status \"fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73\": rpc error: code = NotFound desc = could not find container \"fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73\": container with ID starting with fe8d51687016f28348f9187db536ef8d043c7233285a02c543bd7eb76b754f73 not found: ID does not exist"
Oct 31 02:12:56.337985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.337928    5293 scope.go:122] "RemoveContainer" containerID="c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268"
Oct 31 02:12:56.341907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.341884    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"]
Oct 31 02:12:56.347338 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.347316    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-df9d42-5cff7d5d7-mcjfw"]
Oct 31 02:12:56.355493 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.355428    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"]
Oct 31 02:12:56.355560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.355541    5293 scope.go:122] "RemoveContainer" containerID="c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268"
Oct 31 02:12:56.355868 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:12:56.355838    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268\": container with ID starting with c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268 not found: ID does not exist" containerID="c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268"
Oct 31 02:12:56.356015 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.355877    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268"} err="failed to get container status \"c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268\": rpc error: code = NotFound desc = could not find container \"c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268\": container with ID starting with c5dd0c743924396a98fea1fd2f0561a630fb9c09cfbd9a8b7c01b8ad6c557268 not found: ID does not exist"
Oct 31 02:12:56.360406 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.360385    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-df9d42-675d896bc9-5b9bh"]
Oct 31 02:12:56.375474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.375448    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8lfg\" (UniqueName: \"kubernetes.io/projected/872a4172-8fca-4b53-8bb4-aa0ae3010b58-kube-api-access-t8lfg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:56.375578 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:56.375474    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dvpd\" (UniqueName: \"kubernetes.io/projected/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b-kube-api-access-2dvpd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:58.252564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:58.248225    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" path="/var/lib/kubelet/pods/3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b/volumes"
Oct 31 02:12:58.252564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:58.249398    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872a4172-8fca-4b53-8bb4-aa0ae3010b58" path="/var/lib/kubelet/pods/872a4172-8fca-4b53-8bb4-aa0ae3010b58/volumes"
Oct 31 02:12:59.865014 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:59.864982    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"]
Oct 31 02:12:59.878081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:12:59.878024    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"]
Oct 31 02:13:00.035396 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:00.035363    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"]
Oct 31 02:13:00.042326 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:00.042294    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031b904-kwnpq"]
Oct 31 02:13:00.099552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:00.099511    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:13:00.289065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:00.281911    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c13248-c64a-424b-8312-f4ac5e345345" path="/var/lib/kubelet/pods/95c13248-c64a-424b-8312-f4ac5e345345/volumes"
Oct 31 02:13:02.855115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:02.855071    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"]
Oct 31 02:13:02.861688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:02.861649    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-46a67e-rxc2z"]
Oct 31 02:13:04.178972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:04.178939    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4441c772-db48-4d8a-8734-73483467fe02" path="/var/lib/kubelet/pods/4441c772-db48-4d8a-8734-73483467fe02/volumes"
Oct 31 02:13:04.877063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:04.869116    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g" podUID="0e78db94-ca55-4141-9a34-a5258f5fa0bc" containerName="payload-processing" containerID="cri-o://67f969368eb723c23239757e95e1275ea6b73f2770434a4e71cb53b02daff669" gracePeriod=25
Oct 31 02:13:04.885926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:04.885844    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6" podUID="0f201992-e685-4cbb-8ca9-60b12bd9d550" containerName="payload-pre-processing" containerID="cri-o://4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0" gracePeriod=25
Oct 31 02:13:05.844632 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.844583    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"]
Oct 31 02:13:05.845574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845552    5293 cpu_manager.go:414] "containerMap: removing container" podUID="872a4172-8fca-4b53-8bb4-aa0ae3010b58" containerName="payload-pre-processing"
Oct 31 02:13:05.845574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845573    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872a4172-8fca-4b53-8bb4-aa0ae3010b58" containerName="payload-pre-processing"
Oct 31 02:13:05.845729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845589    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy"
Oct 31 02:13:05.845729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845595    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy"
Oct 31 02:13:05.845729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845611    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" containerName="payload-processing"
Oct 31 02:13:05.845729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845616    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" containerName="payload-processing"
Oct 31 02:13:05.845729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845627    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy"
Oct 31 02:13:05.845729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845632    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy"
Oct 31 02:13:05.845729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845666    5293 cpu_manager.go:414] "containerMap: removing container" podUID="4441c772-db48-4d8a-8734-73483467fe02" containerName="revoke-keys"
Oct 31 02:13:05.845729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845673    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4441c772-db48-4d8a-8734-73483467fe02" containerName="revoke-keys"
Oct 31 02:13:05.846220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845803    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="872a4172-8fca-4b53-8bb4-aa0ae3010b58" containerName="payload-pre-processing"
Oct 31 02:13:05.846220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845813    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy"
Oct 31 02:13:05.846220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845820    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6edfeb-6fb0-4bdd-be9d-6903b9e1da0b" containerName="payload-processing"
Oct 31 02:13:05.846220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845827    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5aa93f-da9b-41e2-acc1-155a54a3767d" containerName="istio-proxy"
Oct 31 02:13:05.846220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.845837    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="4441c772-db48-4d8a-8734-73483467fe02" containerName="revoke-keys"
Oct 31 02:13:05.852299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.852275    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.855556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.855529    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"agc-cred-c008134090fa\""
Oct 31 02:13:05.857837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.857807    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"]
Oct 31 02:13:05.883093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.883007    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.883492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.883458    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.883819 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.883799    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.884069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.884030    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wzj\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-kube-api-access-r4wzj\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.985350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.985317    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.985564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.985475    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.985642 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.985570    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.985642 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.985621    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wzj\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-kube-api-access-r4wzj\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.986583 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.986549    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.986744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.986622    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:05.988464 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:05.988440    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:06.040081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.017789    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wzj\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-kube-api-access-r4wzj\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:06.182110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.182035    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:06.356304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.356271    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"]
Oct 31 02:13:06.363282 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.363258    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.368205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.367865    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-dd41e9644f67\""
Oct 31 02:13:06.368205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.368012    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-36d9b2f51461\""
Oct 31 02:13:06.368205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.368154    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0b6abf\""
Oct 31 02:13:06.372264 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.372235    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"openshift-service-ca.crt\""
Oct 31 02:13:06.372395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.372335    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-e2e--c450cd8d-dockercfg-8zv6g\""
Oct 31 02:13:06.372395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.372383    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"kube-root-ca.crt\""
Oct 31 02:13:06.372573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.372553    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-7b6182fad56e\""
Oct 31 02:13:06.372744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.372729    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"routing-overlay\""
Oct 31 02:13:06.374211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.374183    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"]
Oct 31 02:13:06.392378 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.392346    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.392559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.392508    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qrm\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-kube-api-access-c4qrm\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.392635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.392570    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.392701 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.392656    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.476662 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:06.476627    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdcbbfd_e020_497a_9017_9a623c4f44ba.slice/crio-c6d13d52d47b91951f2f383bbbac3f7ebd8554ea4f7c48d06347b4f6f70edc4e WatchSource:0}: Error finding container c6d13d52d47b91951f2f383bbbac3f7ebd8554ea4f7c48d06347b4f6f70edc4e: Status 404 returned error can't find the container with id c6d13d52d47b91951f2f383bbbac3f7ebd8554ea4f7c48d06347b4f6f70edc4e
Oct 31 02:13:06.478820 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.478794    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"]
Oct 31 02:13:06.494596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.494568    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.494769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.494750    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qrm\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-kube-api-access-c4qrm\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.494837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.494822    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.494941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.494927    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.495743 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.495714    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.495934 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.495908    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.497761 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.497740    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.506267 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.506240    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qrm\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-kube-api-access-c4qrm\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:06.560350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.560309    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj" event={"ID":"bbdcbbfd-e020-497a-9017-9a623c4f44ba","Type":"ContainerStarted","Data":"fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d"}
Oct 31 02:13:06.560514 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.560369    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj" event={"ID":"bbdcbbfd-e020-497a-9017-9a623c4f44ba","Type":"ContainerStarted","Data":"c6d13d52d47b91951f2f383bbbac3f7ebd8554ea4f7c48d06347b4f6f70edc4e"}
Oct 31 02:13:06.560514 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.560428    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:06.583459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.583414    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj" podStartSLOduration=1.583400374 podStartE2EDuration="1.583400374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:06.579090212 +0000 UTC m=+3144.527738600" watchObservedRunningTime="2026-10-31 02:13:06.583400374 +0000 UTC m=+3144.532048781"
Oct 31 02:13:06.692331 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:06.692301    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:07.056901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:07.056871    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"]
Oct 31 02:13:07.057329 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:07.057203    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode043119a_d803_4c53_b7f8_80a2955e0303.slice/crio-ddde7ce931c3512629e653bc73bc110fc5485950a62766b9448ece10ce562ea0 WatchSource:0}: Error finding container ddde7ce931c3512629e653bc73bc110fc5485950a62766b9448ece10ce562ea0: Status 404 returned error can't find the container with id ddde7ce931c3512629e653bc73bc110fc5485950a62766b9448ece10ce562ea0
Oct 31 02:13:07.570453 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:07.570408    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n" event={"ID":"e043119a-d803-4c53-b7f8-80a2955e0303","Type":"ContainerStarted","Data":"ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2"}
Oct 31 02:13:07.570665 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:07.570471    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n" event={"ID":"e043119a-d803-4c53-b7f8-80a2955e0303","Type":"ContainerStarted","Data":"ddde7ce931c3512629e653bc73bc110fc5485950a62766b9448ece10ce562ea0"}
Oct 31 02:13:07.570665 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:07.570507    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:07.594714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:07.594659    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n" podStartSLOduration=1.594645466 podStartE2EDuration="1.594645466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:07.589061588 +0000 UTC m=+3145.537709976" watchObservedRunningTime="2026-10-31 02:13:07.594645466 +0000 UTC m=+3145.543293870"
Oct 31 02:13:08.082185 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:08.082144    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6" podUID="0f201992-e685-4cbb-8ca9-60b12bd9d550" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:13:08.082604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:08.082521    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g" podUID="0e78db94-ca55-4141-9a34-a5258f5fa0bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:13:10.163971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.163932    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"]
Oct 31 02:13:10.164604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.164257    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" containerID="cri-o://d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d" gracePeriod=30
Oct 31 02:13:10.730137 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.727984    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"]
Oct 31 02:13:10.764066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.763249    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.805289 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.805117    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"]
Oct 31 02:13:10.853372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.853324    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:13:10.853729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.853699    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:13:10.854757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.854730    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.854909 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.854872    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86mj\" (UniqueName: \"kubernetes.io/projected/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-kube-api-access-k86mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.855002 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.854928    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.957519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.957488    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.957693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.957562    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86mj\" (UniqueName: \"kubernetes.io/projected/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-kube-api-access-k86mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.957693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.957603    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.958474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.958318    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.960169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.960149    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:10.969008 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:10.968988    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86mj\" (UniqueName: \"kubernetes.io/projected/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-kube-api-access-k86mj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:11.097652 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:11.097542    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:11.406018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:11.405985    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"]
Oct 31 02:13:11.408406 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:11.408374    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb1db4a_abbd_4e9c_bfe4_8b89e70c0856.slice/crio-1def65104365766f645f45985e11d381806ad0ea761ac8f61401d36d88ec066a WatchSource:0}: Error finding container 1def65104365766f645f45985e11d381806ad0ea761ac8f61401d36d88ec066a: Status 404 returned error can't find the container with id 1def65104365766f645f45985e11d381806ad0ea761ac8f61401d36d88ec066a
Oct 31 02:13:11.653239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:11.653197    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd" event={"ID":"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856","Type":"ContainerStarted","Data":"1def65104365766f645f45985e11d381806ad0ea761ac8f61401d36d88ec066a"}
Oct 31 02:13:11.783065 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:11.783014    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:13:12.672104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:12.665235    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd" event={"ID":"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856","Type":"ContainerStarted","Data":"c68c058770487a222ba212b825ab9b08eee8e64e84b72fc0b829c0c9f502bf6c"}
Oct 31 02:13:12.844068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:12.840210    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd" podStartSLOduration=2.840187238 podStartE2EDuration="2.840187238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:12.759109416 +0000 UTC m=+3150.707757823" watchObservedRunningTime="2026-10-31 02:13:12.840187238 +0000 UTC m=+3150.788835631"
Oct 31 02:13:12.910460 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:12.910420    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"]
Oct 31 02:13:12.960114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:12.959757    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:13.193241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.193202    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:13:13.193534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.193504    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:13:13.325121 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.304939    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"]
Oct 31 02:13:13.325121 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.315364    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:13.691554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.687029    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"]
Oct 31 02:13:13.831456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.831432    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:13.831737 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.831709    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"]
Oct 31 02:13:13.851512 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.851474    5293 generic.go:356] "Generic (PLEG): container finished" podID="feb1db4a-abbd-4e9c-bfe4-8b89e70c0856" containerID="c68c058770487a222ba212b825ab9b08eee8e64e84b72fc0b829c0c9f502bf6c" exitCode=0
Oct 31 02:13:13.851641 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.851531    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd" event={"ID":"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856","Type":"ContainerDied","Data":"c68c058770487a222ba212b825ab9b08eee8e64e84b72fc0b829c0c9f502bf6c"}
Oct 31 02:13:13.943364 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.943020    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kb87\" (UniqueName: \"kubernetes.io/projected/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-kube-api-access-2kb87\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:13.943712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.943415    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:13.943712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:13.943643    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:14.046148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.046111    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:14.046336 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.046268    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:14.046502 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.046477    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kb87\" (UniqueName: \"kubernetes.io/projected/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-kube-api-access-2kb87\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:14.046999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.046979    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:14.048822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.048797    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:14.057959 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.057932    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kb87\" (UniqueName: \"kubernetes.io/projected/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-kube-api-access-2kb87\") pod \"maas-api-revoke-sub-bda03711-qdq82\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:14.152587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.152557    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:14.477381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.477353    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"]
Oct 31 02:13:14.477658 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:14.477632    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7682f96_546f_4ef2_8c8e_ef2ce30b34b3.slice/crio-8a732df0bc734640db706f6a86d4f5816f8f8fcd52ad6aa6ef46639a2e6c4c9b WatchSource:0}: Error finding container 8a732df0bc734640db706f6a86d4f5816f8f8fcd52ad6aa6ef46639a2e6c4c9b: Status 404 returned error can't find the container with id 8a732df0bc734640db706f6a86d4f5816f8f8fcd52ad6aa6ef46639a2e6c4c9b
Oct 31 02:13:14.861397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:14.861303    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82" event={"ID":"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3","Type":"ContainerStarted","Data":"8a732df0bc734640db706f6a86d4f5816f8f8fcd52ad6aa6ef46639a2e6c4c9b"}
Oct 31 02:13:15.185515 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.185489    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:15.255246 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.255217    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-openshift-ca\") pod \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") "
Oct 31 02:13:15.255443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.255258    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-xks-ca\") pod \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") "
Oct 31 02:13:15.255443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.255324    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-kube-api-access-k86mj\" (UniqueName: \"kubernetes.io/projected/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-kube-api-access-k86mj\") pod \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\" (UID: \"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856\") "
Oct 31 02:13:15.255650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.255627    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-openshift-ca" pod "feb1db4a-abbd-4e9c-bfe4-8b89e70c0856" (UID: "feb1db4a-abbd-4e9c-bfe4-8b89e70c0856"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:15.256260 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.256188    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.257748 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.257721    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-xks-ca" pod "feb1db4a-abbd-4e9c-bfe4-8b89e70c0856" (UID: "feb1db4a-abbd-4e9c-bfe4-8b89e70c0856"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:15.258288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.258265    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-kube-api-access-k86mj" pod "feb1db4a-abbd-4e9c-bfe4-8b89e70c0856" (UID: "feb1db4a-abbd-4e9c-bfe4-8b89e70c0856"). InnerVolumeSpecName "kube-api-access-k86mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:15.367090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.358555    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86mj\" (UniqueName: \"kubernetes.io/projected/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-kube-api-access-k86mj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.367090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.358582    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.466087 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.466009    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 02:13:15.562381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562350    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-credential-socket\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.562545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562449    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-envoy\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.562545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562472    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-kube-api-access-65xv6\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-kube-api-access-65xv6\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.562545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562507    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-token\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-token\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.562747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562547    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-certs\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.562747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562623    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-credential-socket" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.562852 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562795    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-podinfo\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.562914 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562862    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-certs" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.562914 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562879    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istiod-ca-cert\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.563061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.562961    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-socket\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.563157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.563110    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-data\") pod \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\" (UID: \"20e32b58-f094-4f6c-beec-7ff70cf7efaa\") "
Oct 31 02:13:15.563618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.563378    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-data" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.563618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.563488    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-socket" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.563953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.563925    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istiod-ca-cert" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:15.564354 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.564330    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.564434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.564359    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.564434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.564374    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.564434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.564389    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.564434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.564404    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.564851 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.564827    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-kube-api-access-65xv6" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "kube-api-access-65xv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:15.564907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.564887    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-envoy" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.565083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.565017    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-token" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:15.565199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.565182    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-podinfo" pod "20e32b58-f094-4f6c-beec-7ff70cf7efaa" (UID: "20e32b58-f094-4f6c-beec-7ff70cf7efaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:15.666001 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.665964    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.666001 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.666004    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.666318 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.666019    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65xv6\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-kube-api-access-65xv6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.666318 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.666051    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e32b58-f094-4f6c-beec-7ff70cf7efaa-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.877170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.877033    5293 generic.go:356] "Generic (PLEG): container finished" podID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerID="d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d" exitCode=0
Oct 31 02:13:15.877170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.877136    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"
Oct 31 02:13:15.877170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.877137    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" event={"ID":"20e32b58-f094-4f6c-beec-7ff70cf7efaa","Type":"ContainerDied","Data":"d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d"}
Oct 31 02:13:15.877739 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.877242    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9" event={"ID":"20e32b58-f094-4f6c-beec-7ff70cf7efaa","Type":"ContainerDied","Data":"91974652c04569d91f919710287c3879cb834c2b08b263a36bb7a93ad279579d"}
Oct 31 02:13:15.877739 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.877278    5293 scope.go:122] "RemoveContainer" containerID="d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d"
Oct 31 02:13:15.878964 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.878941    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd" event={"ID":"feb1db4a-abbd-4e9c-bfe4-8b89e70c0856","Type":"ContainerDied","Data":"1def65104365766f645f45985e11d381806ad0ea761ac8f61401d36d88ec066a"}
Oct 31 02:13:15.878964 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.878952    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"
Oct 31 02:13:15.879182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.878986    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1def65104365766f645f45985e11d381806ad0ea761ac8f61401d36d88ec066a"
Oct 31 02:13:15.880951 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.880929    5293 generic.go:356] "Generic (PLEG): container finished" podID="b7682f96-546f-4ef2-8c8e-ef2ce30b34b3" containerID="b1f1ef8f170c6b767e53e731e32bc4f99d15fbf8a39c62ec81375e1c87ca91aa" exitCode=0
Oct 31 02:13:15.881159 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.881014    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82" event={"ID":"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3","Type":"ContainerDied","Data":"b1f1ef8f170c6b767e53e731e32bc4f99d15fbf8a39c62ec81375e1c87ca91aa"}
Oct 31 02:13:15.893355 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.893334    5293 scope.go:122] "RemoveContainer" containerID="d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d"
Oct 31 02:13:15.893620 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:15.893602    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d\": container with ID starting with d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d not found: ID does not exist" containerID="d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d"
Oct 31 02:13:15.893679 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.893629    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d"} err="failed to get container status \"d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d\": rpc error: code = NotFound desc = could not find container \"d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d\": container with ID starting with d65bc4fbf43a6b72b01d18ed73f783acdbf9c8902b504a4455c89cae5e915c9d not found: ID does not exist"
Oct 31 02:13:15.919812 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.919787    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"]
Oct 31 02:13:15.926785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:15.926760    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-j22z9"]
Oct 31 02:13:16.177742 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:16.177714    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" path="/var/lib/kubelet/pods/20e32b58-f094-4f6c-beec-7ff70cf7efaa/volumes"
Oct 31 02:13:17.111147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.111118    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:17.208535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.196125    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-kube-api-access-2kb87\" (UniqueName: \"kubernetes.io/projected/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-kube-api-access-2kb87\") pod \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") "
Oct 31 02:13:17.208535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.196622    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-openshift-ca\") pod \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") "
Oct 31 02:13:17.208535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.196714    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-xks-ca\") pod \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\" (UID: \"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3\") "
Oct 31 02:13:17.271703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.271664    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-openshift-ca" pod "b7682f96-546f-4ef2-8c8e-ef2ce30b34b3" (UID: "b7682f96-546f-4ef2-8c8e-ef2ce30b34b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:17.316142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.310018    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-xks-ca" pod "b7682f96-546f-4ef2-8c8e-ef2ce30b34b3" (UID: "b7682f96-546f-4ef2-8c8e-ef2ce30b34b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:17.316142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.310157    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:17.316142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.311259    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-kube-api-access-2kb87" pod "b7682f96-546f-4ef2-8c8e-ef2ce30b34b3" (UID: "b7682f96-546f-4ef2-8c8e-ef2ce30b34b3"). InnerVolumeSpecName "kube-api-access-2kb87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:17.411931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.411896    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:17.411931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.411928    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kb87\" (UniqueName: \"kubernetes.io/projected/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3-kube-api-access-2kb87\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:17.918066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:17.914118    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj" podUID="bbdcbbfd-e020-497a-9017-9a623c4f44ba" containerName="payload-processing" containerID="cri-o://fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d" gracePeriod=25
Oct 31 02:13:18.036207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.036146    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82" event={"ID":"b7682f96-546f-4ef2-8c8e-ef2ce30b34b3","Type":"ContainerDied","Data":"8a732df0bc734640db706f6a86d4f5816f8f8fcd52ad6aa6ef46639a2e6c4c9b"}
Oct 31 02:13:18.036405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.036303    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a732df0bc734640db706f6a86d4f5816f8f8fcd52ad6aa6ef46639a2e6c4c9b"
Oct 31 02:13:18.036405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.036179    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"
Oct 31 02:13:18.082315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.082018    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g" podUID="0e78db94-ca55-4141-9a34-a5258f5fa0bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:13:18.082630 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.082596    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6" podUID="0f201992-e685-4cbb-8ca9-60b12bd9d550" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:13:18.268608 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.268581    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:18.281244 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.281221    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-plugins-config-volume\") pod \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") "
Oct 31 02:13:18.281408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.281332    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-kube-api-access-r4wzj\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-kube-api-access-r4wzj\") pod \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") "
Oct 31 02:13:18.281559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.281542    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-provider-credentials\") pod \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") "
Oct 31 02:13:18.281640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.281581    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-routing-overlay-volume\") pod \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\" (UID: \"bbdcbbfd-e020-497a-9017-9a623c4f44ba\") "
Oct 31 02:13:18.281640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.281581    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-plugins-config-volume" pod "bbdcbbfd-e020-497a-9017-9a623c4f44ba" (UID: "bbdcbbfd-e020-497a-9017-9a623c4f44ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:18.281985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.281965    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-routing-overlay-volume" pod "bbdcbbfd-e020-497a-9017-9a623c4f44ba" (UID: "bbdcbbfd-e020-497a-9017-9a623c4f44ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:18.282414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.282395    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:18.282483 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.282420    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbdcbbfd-e020-497a-9017-9a623c4f44ba-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:18.284571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.284545    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-provider-credentials" pod "bbdcbbfd-e020-497a-9017-9a623c4f44ba" (UID: "bbdcbbfd-e020-497a-9017-9a623c4f44ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:18.284670 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.284591    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-kube-api-access-r4wzj" pod "bbdcbbfd-e020-497a-9017-9a623c4f44ba" (UID: "bbdcbbfd-e020-497a-9017-9a623c4f44ba"). InnerVolumeSpecName "kube-api-access-r4wzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:18.313098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.309288    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n" podUID="e043119a-d803-4c53-b7f8-80a2955e0303" containerName="payload-processing" containerID="cri-o://ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2" gracePeriod=25
Oct 31 02:13:18.383909 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.383878    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:18.383909 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.383907    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4wzj\" (UniqueName: \"kubernetes.io/projected/bbdcbbfd-e020-497a-9017-9a623c4f44ba-kube-api-access-r4wzj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:18.708907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.708876    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:18.826942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.826686    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-routing-overlay-volume\") pod \"e043119a-d803-4c53-b7f8-80a2955e0303\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") "
Oct 31 02:13:18.826942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.826821    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-plugins-config-volume\") pod \"e043119a-d803-4c53-b7f8-80a2955e0303\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") "
Oct 31 02:13:18.827650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.827150    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-kube-api-access-c4qrm\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-kube-api-access-c4qrm\") pod \"e043119a-d803-4c53-b7f8-80a2955e0303\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") "
Oct 31 02:13:18.827650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.827223    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-provider-credentials\") pod \"e043119a-d803-4c53-b7f8-80a2955e0303\" (UID: \"e043119a-d803-4c53-b7f8-80a2955e0303\") "
Oct 31 02:13:18.827782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.827642    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-routing-overlay-volume" pod "e043119a-d803-4c53-b7f8-80a2955e0303" (UID: "e043119a-d803-4c53-b7f8-80a2955e0303"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:18.827782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.827756    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-plugins-config-volume" pod "e043119a-d803-4c53-b7f8-80a2955e0303" (UID: "e043119a-d803-4c53-b7f8-80a2955e0303"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:18.828266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.828244    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:18.828360 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.828269    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e043119a-d803-4c53-b7f8-80a2955e0303-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:18.829868 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.829847    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-kube-api-access-c4qrm" pod "e043119a-d803-4c53-b7f8-80a2955e0303" (UID: "e043119a-d803-4c53-b7f8-80a2955e0303"). InnerVolumeSpecName "kube-api-access-c4qrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:18.830126 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.830110    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-provider-credentials" pod "e043119a-d803-4c53-b7f8-80a2955e0303" (UID: "e043119a-d803-4c53-b7f8-80a2955e0303"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:18.929768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.929737    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qrm\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-kube-api-access-c4qrm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:18.929768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:18.929767    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e043119a-d803-4c53-b7f8-80a2955e0303-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:19.048230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.048194    5293 generic.go:356] "Generic (PLEG): container finished" podID="e043119a-d803-4c53-b7f8-80a2955e0303" containerID="ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2" exitCode=0
Oct 31 02:13:19.048408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.048243    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"
Oct 31 02:13:19.048408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.048282    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n" event={"ID":"e043119a-d803-4c53-b7f8-80a2955e0303","Type":"ContainerDied","Data":"ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2"}
Oct 31 02:13:19.048408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.048343    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n" event={"ID":"e043119a-d803-4c53-b7f8-80a2955e0303","Type":"ContainerDied","Data":"ddde7ce931c3512629e653bc73bc110fc5485950a62766b9448ece10ce562ea0"}
Oct 31 02:13:19.048408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.048376    5293 scope.go:122] "RemoveContainer" containerID="ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2"
Oct 31 02:13:19.050184 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.050159    5293 generic.go:356] "Generic (PLEG): container finished" podID="bbdcbbfd-e020-497a-9017-9a623c4f44ba" containerID="fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d" exitCode=0
Oct 31 02:13:19.050316 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.050210    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"
Oct 31 02:13:19.050316 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.050222    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj" event={"ID":"bbdcbbfd-e020-497a-9017-9a623c4f44ba","Type":"ContainerDied","Data":"fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d"}
Oct 31 02:13:19.050316 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.050264    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj" event={"ID":"bbdcbbfd-e020-497a-9017-9a623c4f44ba","Type":"ContainerDied","Data":"c6d13d52d47b91951f2f383bbbac3f7ebd8554ea4f7c48d06347b4f6f70edc4e"}
Oct 31 02:13:19.060834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.060818    5293 scope.go:122] "RemoveContainer" containerID="ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2"
Oct 31 02:13:19.061147 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:19.061129    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2\": container with ID starting with ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2 not found: ID does not exist" containerID="ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2"
Oct 31 02:13:19.061215 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.061155    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2"} err="failed to get container status \"ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2\": rpc error: code = NotFound desc = could not find container \"ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2\": container with ID starting with ded97939437a52c4352eb0bc36b16a2ab6b0d5ae474bf8ac67ec44b807e2d5b2 not found: ID does not exist"
Oct 31 02:13:19.061215 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.061172    5293 scope.go:122] "RemoveContainer" containerID="fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d"
Oct 31 02:13:19.072413 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.072386    5293 scope.go:122] "RemoveContainer" containerID="fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d"
Oct 31 02:13:19.072699 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:19.072677    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d\": container with ID starting with fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d not found: ID does not exist" containerID="fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d"
Oct 31 02:13:19.072759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.072705    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d"} err="failed to get container status \"fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d\": rpc error: code = NotFound desc = could not find container \"fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d\": container with ID starting with fcde0a703e2d606aa191b0f1fa956c840dde27e14e74b292ae690c042df5a84d not found: ID does not exist"
Oct 31 02:13:19.075166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.075132    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"]
Oct 31 02:13:19.079571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.079525    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-6868rdq7n"]
Oct 31 02:13:19.089353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.089330    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"]
Oct 31 02:13:19.093575 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:19.093550    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7bdcmbxlj"]
Oct 31 02:13:20.177740 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:20.177708    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbdcbbfd-e020-497a-9017-9a623c4f44ba" path="/var/lib/kubelet/pods/bbdcbbfd-e020-497a-9017-9a623c4f44ba/volumes"
Oct 31 02:13:20.178477 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:20.178452    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e043119a-d803-4c53-b7f8-80a2955e0303" path="/var/lib/kubelet/pods/e043119a-d803-4c53-b7f8-80a2955e0303/volumes"
Oct 31 02:13:23.189679 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:23.189602    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"]
Oct 31 02:13:23.190159 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:23.189824    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy" containerID="cri-o://e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30" gracePeriod=30
Oct 31 02:13:24.085484 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.085449    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"]
Oct 31 02:13:24.086331 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086312    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b7682f96-546f-4ef2-8c8e-ef2ce30b34b3" containerName="revoke-keys"
Oct 31 02:13:24.086331 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086332    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7682f96-546f-4ef2-8c8e-ef2ce30b34b3" containerName="revoke-keys"
Oct 31 02:13:24.086513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086346    5293 cpu_manager.go:414] "containerMap: removing container" podUID="feb1db4a-abbd-4e9c-bfe4-8b89e70c0856" containerName="revoke-keys"
Oct 31 02:13:24.086513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086351    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb1db4a-abbd-4e9c-bfe4-8b89e70c0856" containerName="revoke-keys"
Oct 31 02:13:24.086513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086372    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e043119a-d803-4c53-b7f8-80a2955e0303" containerName="payload-processing"
Oct 31 02:13:24.086513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086378    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e043119a-d803-4c53-b7f8-80a2955e0303" containerName="payload-processing"
Oct 31 02:13:24.086513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086387    5293 cpu_manager.go:414] "containerMap: removing container" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy"
Oct 31 02:13:24.086513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086392    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy"
Oct 31 02:13:24.086513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086413    5293 cpu_manager.go:414] "containerMap: removing container" podUID="bbdcbbfd-e020-497a-9017-9a623c4f44ba" containerName="payload-processing"
Oct 31 02:13:24.086513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086418    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdcbbfd-e020-497a-9017-9a623c4f44ba" containerName="payload-processing"
Oct 31 02:13:24.086922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086543    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdcbbfd-e020-497a-9017-9a623c4f44ba" containerName="payload-processing"
Oct 31 02:13:24.086922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086555    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7682f96-546f-4ef2-8c8e-ef2ce30b34b3" containerName="revoke-keys"
Oct 31 02:13:24.086922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086563    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb1db4a-abbd-4e9c-bfe4-8b89e70c0856" containerName="revoke-keys"
Oct 31 02:13:24.086922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086574    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e043119a-d803-4c53-b7f8-80a2955e0303" containerName="payload-processing"
Oct 31 02:13:24.086922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.086580    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e32b58-f094-4f6c-beec-7ff70cf7efaa" containerName="istio-proxy"
Oct 31 02:13:24.091891 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.091872    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.096848 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.096826    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:13:24.097080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.096872    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:13:24.103783 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.102947    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"]
Oct 31 02:13:24.262422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.222885    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zzp\" (UniqueName: \"kubernetes.io/projected/5163832d-414a-424c-8c48-0ba095d76b12-kube-api-access-t7zzp\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.262422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.222994    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.262422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.223227    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.389122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.370351    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.415473 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.402402    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zzp\" (UniqueName: \"kubernetes.io/projected/5163832d-414a-424c-8c48-0ba095d76b12-kube-api-access-t7zzp\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.415473 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.402537    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.433178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.416664    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.507674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.498489    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.579579 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.579170    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zzp\" (UniqueName: \"kubernetes.io/projected/5163832d-414a-424c-8c48-0ba095d76b12-kube-api-access-t7zzp\") pod \"maas-api-revoke-sub-dfb99a85-vw5z2\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:24.719288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:24.719257    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:25.234392 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.234147    5293 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:13:25.256132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.255963    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:13:25.306856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.298551    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:13:25.541782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.541651    5293 generic.go:356] "Generic (PLEG): container finished" podID="0e78db94-ca55-4141-9a34-a5258f5fa0bc" containerID="67f969368eb723c23239757e95e1275ea6b73f2770434a4e71cb53b02daff669" exitCode=0
Oct 31 02:13:25.541963 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.541818    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g" event={"ID":"0e78db94-ca55-4141-9a34-a5258f5fa0bc","Type":"ContainerDied","Data":"67f969368eb723c23239757e95e1275ea6b73f2770434a4e71cb53b02daff669"}
Oct 31 02:13:25.544126 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.543897    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 02:13:25.544634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.544609    5293 generic.go:356] "Generic (PLEG): container finished" podID="0f201992-e685-4cbb-8ca9-60b12bd9d550" containerID="4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0" exitCode=0
Oct 31 02:13:25.544761 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.544679    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6" event={"ID":"0f201992-e685-4cbb-8ca9-60b12bd9d550","Type":"ContainerDied","Data":"4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0"}
Oct 31 02:13:25.544761 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.544730    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6" event={"ID":"0f201992-e685-4cbb-8ca9-60b12bd9d550","Type":"ContainerDied","Data":"443f36381d10f33968258e27b75ac6e0bba58b4887cf7c0b9869b681be3de9db"}
Oct 31 02:13:25.544890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.544763    5293 scope.go:122] "RemoveContainer" containerID="4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0"
Oct 31 02:13:25.572497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.571983    5293 scope.go:122] "RemoveContainer" containerID="4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0"
Oct 31 02:13:25.572497 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:25.572393    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0\": container with ID starting with 4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0 not found: ID does not exist" containerID="4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0"
Oct 31 02:13:25.572497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.572429    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0"} err="failed to get container status \"4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0\": rpc error: code = NotFound desc = could not find container \"4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0\": container with ID starting with 4d3fb3cb642a1b83bc1e312f064f670095844ab11cc37d756212eadbb9bc8da0 not found: ID does not exist"
Oct 31 02:13:25.697289 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.697256    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f201992-e685-4cbb-8ca9-60b12bd9d550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f201992-e685-4cbb-8ca9-60b12bd9d550-plugins-config-volume\") pod \"0f201992-e685-4cbb-8ca9-60b12bd9d550\" (UID: \"0f201992-e685-4cbb-8ca9-60b12bd9d550\") "
Oct 31 02:13:25.697442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.697382    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f201992-e685-4cbb-8ca9-60b12bd9d550-kube-api-access-8vm7p\" (UniqueName: \"kubernetes.io/projected/0f201992-e685-4cbb-8ca9-60b12bd9d550-kube-api-access-8vm7p\") pod \"0f201992-e685-4cbb-8ca9-60b12bd9d550\" (UID: \"0f201992-e685-4cbb-8ca9-60b12bd9d550\") "
Oct 31 02:13:25.697791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.697751    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f201992-e685-4cbb-8ca9-60b12bd9d550-plugins-config-volume" pod "0f201992-e685-4cbb-8ca9-60b12bd9d550" (UID: "0f201992-e685-4cbb-8ca9-60b12bd9d550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:25.698835 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.698808    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f201992-e685-4cbb-8ca9-60b12bd9d550-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.700026 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.700001    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f201992-e685-4cbb-8ca9-60b12bd9d550-kube-api-access-8vm7p" pod "0f201992-e685-4cbb-8ca9-60b12bd9d550" (UID: "0f201992-e685-4cbb-8ca9-60b12bd9d550"). InnerVolumeSpecName "kube-api-access-8vm7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:25.725830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.725800    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 02:13:25.858869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.807995    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vm7p\" (UniqueName: \"kubernetes.io/projected/0f201992-e685-4cbb-8ca9-60b12bd9d550-kube-api-access-8vm7p\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.858869 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:25.856675    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5163832d_414a_424c_8c48_0ba095d76b12.slice/crio-e6d7ef7da855b0ea18919737de2356a1efabff0bfb145c2e3b05d2cc4bc50e14 WatchSource:0}: Error finding container e6d7ef7da855b0ea18919737de2356a1efabff0bfb145c2e3b05d2cc4bc50e14: Status 404 returned error can't find the container with id e6d7ef7da855b0ea18919737de2356a1efabff0bfb145c2e3b05d2cc4bc50e14
Oct 31 02:13:25.874136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.874104    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"]
Oct 31 02:13:25.909111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.909081    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e78db94-ca55-4141-9a34-a5258f5fa0bc-kube-api-access-n5lqq\" (UniqueName: \"kubernetes.io/projected/0e78db94-ca55-4141-9a34-a5258f5fa0bc-kube-api-access-n5lqq\") pod \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\" (UID: \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\") "
Oct 31 02:13:25.909243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.909209    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e78db94-ca55-4141-9a34-a5258f5fa0bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78db94-ca55-4141-9a34-a5258f5fa0bc-plugins-config-volume\") pod \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\" (UID: \"0e78db94-ca55-4141-9a34-a5258f5fa0bc\") "
Oct 31 02:13:25.910573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.910545    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e78db94-ca55-4141-9a34-a5258f5fa0bc-plugins-config-volume" pod "0e78db94-ca55-4141-9a34-a5258f5fa0bc" (UID: "0e78db94-ca55-4141-9a34-a5258f5fa0bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:25.912806 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:25.912778    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e78db94-ca55-4141-9a34-a5258f5fa0bc-kube-api-access-n5lqq" pod "0e78db94-ca55-4141-9a34-a5258f5fa0bc" (UID: "0e78db94-ca55-4141-9a34-a5258f5fa0bc"). InnerVolumeSpecName "kube-api-access-n5lqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:26.012360 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.012323    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lqq\" (UniqueName: \"kubernetes.io/projected/0e78db94-ca55-4141-9a34-a5258f5fa0bc-kube-api-access-n5lqq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:26.012360 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.012351    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78db94-ca55-4141-9a34-a5258f5fa0bc-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:26.558141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.557736    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g" event={"ID":"0e78db94-ca55-4141-9a34-a5258f5fa0bc","Type":"ContainerDied","Data":"9fda3c4343d996941842f25a53fab3a503114577ab77e910be11a30588435c30"}
Oct 31 02:13:26.558141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.557825    5293 scope.go:122] "RemoveContainer" containerID="67f969368eb723c23239757e95e1275ea6b73f2770434a4e71cb53b02daff669"
Oct 31 02:13:26.558647 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.558474    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"
Oct 31 02:13:26.559780 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.559754    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"
Oct 31 02:13:26.565188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.563392    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2" event={"ID":"5163832d-414a-424c-8c48-0ba095d76b12","Type":"ContainerStarted","Data":"a65cb507b1adfefd6a46decf4b7b71e78ca857629c441cac281e450815f4ba11"}
Oct 31 02:13:26.565188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.563444    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2" event={"ID":"5163832d-414a-424c-8c48-0ba095d76b12","Type":"ContainerStarted","Data":"e6d7ef7da855b0ea18919737de2356a1efabff0bfb145c2e3b05d2cc4bc50e14"}
Oct 31 02:13:26.592992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.592950    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2" podStartSLOduration=2.592937296 podStartE2EDuration="2.592937296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:26.589217265 +0000 UTC m=+3164.537865657" watchObservedRunningTime="2026-10-31 02:13:26.592937296 +0000 UTC m=+3164.541585688"
Oct 31 02:13:26.605620 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.605589    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"]
Oct 31 02:13:26.610210 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.610183    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-46a67e-59854dc8dc-bnd5g"]
Oct 31 02:13:26.622349 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.622318    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"]
Oct 31 02:13:26.626609 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:26.626584    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-46a67e-776d585d75-rf4s6"]
Oct 31 02:13:27.573360 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:27.573328    5293 generic.go:356] "Generic (PLEG): container finished" podID="5163832d-414a-424c-8c48-0ba095d76b12" containerID="a65cb507b1adfefd6a46decf4b7b71e78ca857629c441cac281e450815f4ba11" exitCode=0
Oct 31 02:13:27.573755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:27.573403    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2" event={"ID":"5163832d-414a-424c-8c48-0ba095d76b12","Type":"ContainerDied","Data":"a65cb507b1adfefd6a46decf4b7b71e78ca857629c441cac281e450815f4ba11"}
Oct 31 02:13:28.177920 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.177887    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e78db94-ca55-4141-9a34-a5258f5fa0bc" path="/var/lib/kubelet/pods/0e78db94-ca55-4141-9a34-a5258f5fa0bc/volumes"
Oct 31 02:13:28.178423 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.178406    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f201992-e685-4cbb-8ca9-60b12bd9d550" path="/var/lib/kubelet/pods/0f201992-e685-4cbb-8ca9-60b12bd9d550/volumes"
Oct 31 02:13:28.587303 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.587272    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:13:28.597450 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.597417    5293 generic.go:356] "Generic (PLEG): container finished" podID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerID="e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30" exitCode=0
Oct 31 02:13:28.598160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.597721    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"
Oct 31 02:13:28.598160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.597911    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" event={"ID":"9afd0a65-9e74-4835-a35a-72b63c08c7a8","Type":"ContainerDied","Data":"e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30"}
Oct 31 02:13:28.598160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.597955    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh" event={"ID":"9afd0a65-9e74-4835-a35a-72b63c08c7a8","Type":"ContainerDied","Data":"7731323cc4b5c02dc6eea75968492504f8a740e836fb237e406c4d99537b1501"}
Oct 31 02:13:28.598160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.597985    5293 scope.go:122] "RemoveContainer" containerID="e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30"
Oct 31 02:13:28.604790 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.603624    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-credential-socket\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.604790 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.603672    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-socket\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.604790 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.603694    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-kube-api-access-fk9w5\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-kube-api-access-fk9w5\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.604790 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.603740    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-envoy\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.604790 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.603799    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-certs\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.604790 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.603834    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-podinfo\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.604790 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.604667    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-credential-socket" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:28.605108 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.605035    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-socket" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:28.605358 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.605265    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-certs" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:28.607407 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.607267    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-kube-api-access-fk9w5" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "kube-api-access-fk9w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:28.607407 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.607370    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-envoy" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:28.611088 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.608918    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-podinfo" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:28.639199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.635373    5293 scope.go:122] "RemoveContainer" containerID="e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30"
Oct 31 02:13:28.639199 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:28.635854    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30\": container with ID starting with e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30 not found: ID does not exist" containerID="e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30"
Oct 31 02:13:28.639199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.635893    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30"} err="failed to get container status \"e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30\": rpc error: code = NotFound desc = could not find container \"e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30\": container with ID starting with e068bc4600b5fe64f1aa70163ac4dc148082ef10336aafc5f9d579a23d992d30 not found: ID does not exist"
Oct 31 02:13:28.706165 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.706124    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istiod-ca-cert\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.707239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.707139    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istiod-ca-cert" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:28.707583 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.707560    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-token\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.708518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.708500    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-data\") pod \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\" (UID: \"9afd0a65-9e74-4835-a35a-72b63c08c7a8\") "
Oct 31 02:13:28.709086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.708979    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-data" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:28.710098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.710019    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.710235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.710218    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.710336 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.710322    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9w5\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-kube-api-access-fk9w5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.710432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.710419    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.710536 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.710522    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.710637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.710623    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.710734 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.710722    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.710822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.710810    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.714012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.713978    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-token" pod "9afd0a65-9e74-4835-a35a-72b63c08c7a8" (UID: "9afd0a65-9e74-4835-a35a-72b63c08c7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:28.800136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.800106    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:28.811798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.811768    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-xks-ca\") pod \"5163832d-414a-424c-8c48-0ba095d76b12\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") "
Oct 31 02:13:28.811968 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.811911    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-openshift-ca\") pod \"5163832d-414a-424c-8c48-0ba095d76b12\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") "
Oct 31 02:13:28.812062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.811995    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5163832d-414a-424c-8c48-0ba095d76b12-kube-api-access-t7zzp\" (UniqueName: \"kubernetes.io/projected/5163832d-414a-424c-8c48-0ba095d76b12-kube-api-access-t7zzp\") pod \"5163832d-414a-424c-8c48-0ba095d76b12\" (UID: \"5163832d-414a-424c-8c48-0ba095d76b12\") "
Oct 31 02:13:28.812328 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.812299    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-openshift-ca" pod "5163832d-414a-424c-8c48-0ba095d76b12" (UID: "5163832d-414a-424c-8c48-0ba095d76b12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:28.812886 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.812862    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.812982 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.812891    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afd0a65-9e74-4835-a35a-72b63c08c7a8-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.814263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.814234    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5163832d-414a-424c-8c48-0ba095d76b12-kube-api-access-t7zzp" pod "5163832d-414a-424c-8c48-0ba095d76b12" (UID: "5163832d-414a-424c-8c48-0ba095d76b12"). InnerVolumeSpecName "kube-api-access-t7zzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:28.814263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.814246    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-xks-ca" pod "5163832d-414a-424c-8c48-0ba095d76b12" (UID: "5163832d-414a-424c-8c48-0ba095d76b12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:28.914343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.914314    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5163832d-414a-424c-8c48-0ba095d76b12-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.914343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.914340    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zzp\" (UniqueName: \"kubernetes.io/projected/5163832d-414a-424c-8c48-0ba095d76b12-kube-api-access-t7zzp\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:28.923945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.923916    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"]
Oct 31 02:13:28.932737 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:28.932715    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-9j4dh"]
Oct 31 02:13:29.607913 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:29.607829    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"
Oct 31 02:13:29.607913 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:29.607856    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2" event={"ID":"5163832d-414a-424c-8c48-0ba095d76b12","Type":"ContainerDied","Data":"e6d7ef7da855b0ea18919737de2356a1efabff0bfb145c2e3b05d2cc4bc50e14"}
Oct 31 02:13:29.607913 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:29.607913    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6d7ef7da855b0ea18919737de2356a1efabff0bfb145c2e3b05d2cc4bc50e14"
Oct 31 02:13:30.181465 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:30.181426    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" path="/var/lib/kubelet/pods/9afd0a65-9e74-4835-a35a-72b63c08c7a8/volumes"
Oct 31 02:13:30.817064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:30.816553    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"]
Oct 31 02:13:30.846065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:30.842641    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-6c54a1-sssqd"]
Oct 31 02:13:32.181534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:32.181498    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb1db4a-abbd-4e9c-bfe4-8b89e70c0856" path="/var/lib/kubelet/pods/feb1db4a-abbd-4e9c-bfe4-8b89e70c0856/volumes"
Oct 31 02:13:34.784574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:34.784543    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"]
Oct 31 02:13:34.785030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:34.784757    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy" containerID="cri-o://2142d3f0f394839a8e618233859f769c0df786f17a8f39454cd315747b2b8966" gracePeriod=30
Oct 31 02:13:34.789115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:34.789087    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"]
Oct 31 02:13:34.789585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:34.789554    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy" containerID="cri-o://2cb1a8a685caea5438f733adf141ef4c3db961a4c487f1489d4ef2208afb7bd5" gracePeriod=30
Oct 31 02:13:35.520788 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.520754    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"]
Oct 31 02:13:35.521986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.521959    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0e78db94-ca55-4141-9a34-a5258f5fa0bc" containerName="payload-processing"
Oct 31 02:13:35.521986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.521986    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e78db94-ca55-4141-9a34-a5258f5fa0bc" containerName="payload-processing"
Oct 31 02:13:35.522188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522004    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy"
Oct 31 02:13:35.522188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522012    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy"
Oct 31 02:13:35.522188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522027    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5163832d-414a-424c-8c48-0ba095d76b12" containerName="revoke-keys"
Oct 31 02:13:35.522188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522034    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5163832d-414a-424c-8c48-0ba095d76b12" containerName="revoke-keys"
Oct 31 02:13:35.522188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522072    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0f201992-e685-4cbb-8ca9-60b12bd9d550" containerName="payload-pre-processing"
Oct 31 02:13:35.522188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522079    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f201992-e685-4cbb-8ca9-60b12bd9d550" containerName="payload-pre-processing"
Oct 31 02:13:35.522518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522295    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e78db94-ca55-4141-9a34-a5258f5fa0bc" containerName="payload-processing"
Oct 31 02:13:35.522518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522310    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afd0a65-9e74-4835-a35a-72b63c08c7a8" containerName="istio-proxy"
Oct 31 02:13:35.522518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522321    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5163832d-414a-424c-8c48-0ba095d76b12" containerName="revoke-keys"
Oct 31 02:13:35.522518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.522333    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f201992-e685-4cbb-8ca9-60b12bd9d550" containerName="payload-pre-processing"
Oct 31 02:13:35.598220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.579359    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.635078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.629405    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"]
Oct 31 02:13:35.759978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.759927    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.759978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.759980    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.760390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.760144    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.760390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.760202    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw447\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-kube-api-access-gw447\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.861797 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.861712    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.861797 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.861762    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.862459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.861858    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.862459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.861901    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw447\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-kube-api-access-gw447\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.862597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.862569    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.862743 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.862711    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.864833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.864812    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.874638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.874615    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw447\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-kube-api-access-gw447\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:35.979213 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.979182    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"]
Oct 31 02:13:35.984280 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.984257    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:35.987900 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.987877    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-36d9b2f51461\""
Oct 31 02:13:35.989078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.988875    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-e2e--c450cd8d-dockercfg-8zv6g\""
Oct 31 02:13:35.989078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.988895    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"kube-root-ca.crt\""
Oct 31 02:13:35.989078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.988953    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"openshift-service-ca.crt\""
Oct 31 02:13:35.989078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.988955    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"routing-overlay\""
Oct 31 02:13:35.989078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.988989    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0b6abf\""
Oct 31 02:13:35.996191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:35.996153    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"]
Oct 31 02:13:36.096030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.095938    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:36.260089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.199991    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.400218 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.387837    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzbx6\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-kube-api-access-pzbx6\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.400218 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.387961    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.400218 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.388065    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.463729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.457903    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"]
Oct 31 02:13:36.468601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.468404    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"]
Oct 31 02:13:36.468601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.468522    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.472005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.471958    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:13:36.472202 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.472171    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:13:36.489686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.489645    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjz9\" (UniqueName: \"kubernetes.io/projected/46aed14a-2163-408d-bb68-fa6951479193-kube-api-access-6qjz9\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.489900 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.489759    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aed14a-2163-408d-bb68-fa6951479193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.490084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.490058    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aed14a-2163-408d-bb68-fa6951479193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.490249 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.490225    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.490318 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.490278    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzbx6\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-kube-api-access-pzbx6\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.490374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.490332    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.490441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.490373    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.491026 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.491002    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.511275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.491643    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.511275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.493974    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.537123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.525092    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzbx6\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-kube-api-access-pzbx6\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.599112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.594788    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjz9\" (UniqueName: \"kubernetes.io/projected/46aed14a-2163-408d-bb68-fa6951479193-kube-api-access-6qjz9\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.599112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.594859    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aed14a-2163-408d-bb68-fa6951479193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.599112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.596156    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aed14a-2163-408d-bb68-fa6951479193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.599112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.597515    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aed14a-2163-408d-bb68-fa6951479193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.603210 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.601062    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aed14a-2163-408d-bb68-fa6951479193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.612062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.610970    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:36.654363 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.653784    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjz9\" (UniqueName: \"kubernetes.io/projected/46aed14a-2163-408d-bb68-fa6951479193-kube-api-access-6qjz9\") pod \"maas-api-revoke-sub-5a2835ee-24z6t\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.861119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.861083    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:36.877933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.877910    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"]
Oct 31 02:13:36.881364 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:36.881333    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c566c7_8fa5_4e50_80e0_127bd356d02d.slice/crio-27231d23028d10ce51529e124a3bf3a6471d31a1a3bd25e34cee28867e59ff19 WatchSource:0}: Error finding container 27231d23028d10ce51529e124a3bf3a6471d31a1a3bd25e34cee28867e59ff19: Status 404 returned error can't find the container with id 27231d23028d10ce51529e124a3bf3a6471d31a1a3bd25e34cee28867e59ff19
Oct 31 02:13:36.952386 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:36.950063    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"]
Oct 31 02:13:36.977027 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:36.976114    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b6bee3_6b9b_402f_b18a_43418d9ce3ff.slice/crio-293d07e85d93bb5b4c8f8130dea95e9a17d7610a488ff2a44ab2beebb75c015c WatchSource:0}: Error finding container 293d07e85d93bb5b4c8f8130dea95e9a17d7610a488ff2a44ab2beebb75c015c: Status 404 returned error can't find the container with id 293d07e85d93bb5b4c8f8130dea95e9a17d7610a488ff2a44ab2beebb75c015c
Oct 31 02:13:37.158259 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.158231    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"]
Oct 31 02:13:37.158793 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:37.158768    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46aed14a_2163_408d_bb68_fa6951479193.slice/crio-33abd207c44788dc7f8eac9a2d09310495adcf2349bc521d32ef1bb561fed9a3 WatchSource:0}: Error finding container 33abd207c44788dc7f8eac9a2d09310495adcf2349bc521d32ef1bb561fed9a3: Status 404 returned error can't find the container with id 33abd207c44788dc7f8eac9a2d09310495adcf2349bc521d32ef1bb561fed9a3
Oct 31 02:13:37.816143 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.816012    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t" event={"ID":"46aed14a-2163-408d-bb68-fa6951479193","Type":"ContainerStarted","Data":"3114a6f9b64a35f63d877bfe30c44d046d3ee00640bea4c372f09f541f1a5743"}
Oct 31 02:13:37.816143 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.816133    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t" event={"ID":"46aed14a-2163-408d-bb68-fa6951479193","Type":"ContainerStarted","Data":"33abd207c44788dc7f8eac9a2d09310495adcf2349bc521d32ef1bb561fed9a3"}
Oct 31 02:13:37.820564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.819814    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8" event={"ID":"a2c566c7-8fa5-4e50-80e0-127bd356d02d","Type":"ContainerStarted","Data":"2322b1956836fb4cb689825b0e09944caace83b5524dedb9e356598306488584"}
Oct 31 02:13:37.820564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.819876    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8" event={"ID":"a2c566c7-8fa5-4e50-80e0-127bd356d02d","Type":"ContainerStarted","Data":"27231d23028d10ce51529e124a3bf3a6471d31a1a3bd25e34cee28867e59ff19"}
Oct 31 02:13:37.820564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.820388    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:37.822121 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.822099    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp" event={"ID":"84b6bee3-6b9b-402f-b18a-43418d9ce3ff","Type":"ContainerStarted","Data":"de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3"}
Oct 31 02:13:37.822238 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.822142    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp" event={"ID":"84b6bee3-6b9b-402f-b18a-43418d9ce3ff","Type":"ContainerStarted","Data":"293d07e85d93bb5b4c8f8130dea95e9a17d7610a488ff2a44ab2beebb75c015c"}
Oct 31 02:13:37.822238 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.822187    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:37.839119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.839074    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t" podStartSLOduration=1.839061254 podStartE2EDuration="1.839061254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:37.834197464 +0000 UTC m=+3175.782845858" watchObservedRunningTime="2026-10-31 02:13:37.839061254 +0000 UTC m=+3175.787709646"
Oct 31 02:13:37.853450 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.853407    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp" podStartSLOduration=2.8533927500000003 podStartE2EDuration="2.85339275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:37.852018758 +0000 UTC m=+3175.800667150" watchObservedRunningTime="2026-10-31 02:13:37.85339275 +0000 UTC m=+3175.802041143"
Oct 31 02:13:37.874769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:37.874717    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8" podStartSLOduration=2.874703049 podStartE2EDuration="2.874703049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:37.871407928 +0000 UTC m=+3175.820056319" watchObservedRunningTime="2026-10-31 02:13:37.874703049 +0000 UTC m=+3175.823351470"
Oct 31 02:13:38.923850 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:38.923800    5293 generic.go:356] "Generic (PLEG): container finished" podID="46aed14a-2163-408d-bb68-fa6951479193" containerID="3114a6f9b64a35f63d877bfe30c44d046d3ee00640bea4c372f09f541f1a5743" exitCode=0
Oct 31 02:13:38.924777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:38.924275    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t" event={"ID":"46aed14a-2163-408d-bb68-fa6951479193","Type":"ContainerDied","Data":"3114a6f9b64a35f63d877bfe30c44d046d3ee00640bea4c372f09f541f1a5743"}
Oct 31 02:13:39.081068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.078603    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"]
Oct 31 02:13:39.086255 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.086223    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.089884 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.089857    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"]
Oct 31 02:13:39.180898 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.180806    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.193157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.181915    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cvnk\" (UniqueName: \"kubernetes.io/projected/0f264f44-9b2a-405f-bb7a-62e769613363-kube-api-access-6cvnk\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.193157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.182096    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.283065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.283019    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.283270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.283252    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cvnk\" (UniqueName: \"kubernetes.io/projected/0f264f44-9b2a-405f-bb7a-62e769613363-kube-api-access-6cvnk\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.283342 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.283314    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.283832 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.283808    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.286230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.286206    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.295593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.295570    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cvnk\" (UniqueName: \"kubernetes.io/projected/0f264f44-9b2a-405f-bb7a-62e769613363-kube-api-access-6cvnk\") pod \"maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.420517 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.420479    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:39.716881 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:39.716848    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"]
Oct 31 02:13:39.717437 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:13:39.717416    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f264f44_9b2a_405f_bb7a_62e769613363.slice/crio-e42c46ef969f34080a91da146e1fcd1be1f1f714d84a91c0bea5d9bcfd4be837 WatchSource:0}: Error finding container e42c46ef969f34080a91da146e1fcd1be1f1f714d84a91c0bea5d9bcfd4be837: Status 404 returned error can't find the container with id e42c46ef969f34080a91da146e1fcd1be1f1f714d84a91c0bea5d9bcfd4be837
Oct 31 02:13:40.069932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.069125    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx" event={"ID":"0f264f44-9b2a-405f-bb7a-62e769613363","Type":"ContainerStarted","Data":"e42c46ef969f34080a91da146e1fcd1be1f1f714d84a91c0bea5d9bcfd4be837"}
Oct 31 02:13:40.072831 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.072240    5293 generic.go:356] "Generic (PLEG): container finished" podID="87139ff5-903f-424b-9e99-b8740dc075e2" containerID="2142d3f0f394839a8e618233859f769c0df786f17a8f39454cd315747b2b8966" exitCode=0
Oct 31 02:13:40.072831 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.072350    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" event={"ID":"87139ff5-903f-424b-9e99-b8740dc075e2","Type":"ContainerDied","Data":"2142d3f0f394839a8e618233859f769c0df786f17a8f39454cd315747b2b8966"}
Oct 31 02:13:40.074685 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.074657    5293 generic.go:356] "Generic (PLEG): container finished" podID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerID="2cb1a8a685caea5438f733adf141ef4c3db961a4c487f1489d4ef2208afb7bd5" exitCode=0
Oct 31 02:13:40.074985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.074728    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" event={"ID":"9cdf2c36-fc81-44e6-b79b-4b6ccd770189","Type":"ContainerDied","Data":"2cb1a8a685caea5438f733adf141ef4c3db961a4c487f1489d4ef2208afb7bd5"}
Oct 31 02:13:40.195688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.195643    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:13:40.248625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.248505    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-istio-token\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-istio-token\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.249090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.249067    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-kube-api-access-52k2n\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-kube-api-access-52k2n\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.249482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.249465    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-envoy\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.249663 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.249648    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-socket\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.249985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.249954    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-credential-socket\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.250535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.250517    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87139ff5-903f-424b-9e99-b8740dc075e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87139ff5-903f-424b-9e99-b8740dc075e2-istiod-ca-cert\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.253085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.253067    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-data\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.253319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.253304    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-certs\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.253430 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.253417    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87139ff5-903f-424b-9e99-b8740dc075e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87139ff5-903f-424b-9e99-b8740dc075e2-istio-podinfo\") pod \"87139ff5-903f-424b-9e99-b8740dc075e2\" (UID: \"87139ff5-903f-424b-9e99-b8740dc075e2\") "
Oct 31 02:13:40.256298 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.252276    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-socket" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.256428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.255596    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-certs" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.256545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.255766    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-credential-socket" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.256852 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.255921    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87139ff5-903f-424b-9e99-b8740dc075e2-istiod-ca-cert" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:40.257062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.256250    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-data" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.257062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.256766    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-istio-token" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:40.257596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.257575    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-kube-api-access-52k2n" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "kube-api-access-52k2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:40.257988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.257974    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.258221 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.258208    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.258345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.258331    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.258471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.258460    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.258574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.258562    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.258676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.258662    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87139ff5-903f-424b-9e99-b8740dc075e2-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.259992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.259957    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87139ff5-903f-424b-9e99-b8740dc075e2-istio-podinfo" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:40.262102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.262084    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-envoy" pod "87139ff5-903f-424b-9e99-b8740dc075e2" (UID: "87139ff5-903f-424b-9e99-b8740dc075e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.381100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.368345    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87139ff5-903f-424b-9e99-b8740dc075e2-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.381100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.368373    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52k2n\" (UniqueName: \"kubernetes.io/projected/87139ff5-903f-424b-9e99-b8740dc075e2-kube-api-access-52k2n\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.381100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.368389    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87139ff5-903f-424b-9e99-b8740dc075e2-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.381100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.380578    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:13:40.438482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.438454    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:40.469425 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469396    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-data\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.469627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469452    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istiod-ca-cert\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.469627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469479    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-socket\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.469627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469541    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-envoy\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.469794 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469633    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-certs\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.469794 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469721    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-data" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.470033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.470006    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istiod-ca-cert" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:40.470033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469747    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-kube-api-access-8nhwz\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-kube-api-access-8nhwz\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.470171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469854    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-socket" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.470171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.469994    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-certs" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.470171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.470142    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-token\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-token\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.470291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.470193    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-podinfo\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.470332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.470308    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-credential-socket\") pod \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\" (UID: \"9cdf2c36-fc81-44e6-b79b-4b6ccd770189\") "
Oct 31 02:13:40.470750 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.470723    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-credential-socket" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.471142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.471115    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.471142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.471134    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.471142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.471145    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.471363 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.471154    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.471363 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.471172    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.472692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.472654    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-envoy" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:40.472815 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.472772    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-podinfo" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:40.473090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.473067    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-kube-api-access-8nhwz" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "kube-api-access-8nhwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:40.474020 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.473976    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-token" pod "9cdf2c36-fc81-44e6-b79b-4b6ccd770189" (UID: "9cdf2c36-fc81-44e6-b79b-4b6ccd770189"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:40.572515 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.572428    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46aed14a-2163-408d-bb68-fa6951479193-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aed14a-2163-408d-bb68-fa6951479193-maas-api-openshift-ca\") pod \"46aed14a-2163-408d-bb68-fa6951479193\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") "
Oct 31 02:13:40.572686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.572616    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46aed14a-2163-408d-bb68-fa6951479193-kube-api-access-6qjz9\" (UniqueName: \"kubernetes.io/projected/46aed14a-2163-408d-bb68-fa6951479193-kube-api-access-6qjz9\") pod \"46aed14a-2163-408d-bb68-fa6951479193\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") "
Oct 31 02:13:40.572767 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.572743    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46aed14a-2163-408d-bb68-fa6951479193-maas-api-openshift-ca" pod "46aed14a-2163-408d-bb68-fa6951479193" (UID: "46aed14a-2163-408d-bb68-fa6951479193"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:40.572809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.572773    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46aed14a-2163-408d-bb68-fa6951479193-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aed14a-2163-408d-bb68-fa6951479193-maas-api-xks-ca\") pod \"46aed14a-2163-408d-bb68-fa6951479193\" (UID: \"46aed14a-2163-408d-bb68-fa6951479193\") "
Oct 31 02:13:40.573380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.573359    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.573380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.573381    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aed14a-2163-408d-bb68-fa6951479193-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.573380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.573391    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nhwz\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-kube-api-access-8nhwz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.573581 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.573405    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.573581 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.573417    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdf2c36-fc81-44e6-b79b-4b6ccd770189-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.574984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.574962    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46aed14a-2163-408d-bb68-fa6951479193-maas-api-xks-ca" pod "46aed14a-2163-408d-bb68-fa6951479193" (UID: "46aed14a-2163-408d-bb68-fa6951479193"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:40.575119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.574991    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46aed14a-2163-408d-bb68-fa6951479193-kube-api-access-6qjz9" pod "46aed14a-2163-408d-bb68-fa6951479193" (UID: "46aed14a-2163-408d-bb68-fa6951479193"). InnerVolumeSpecName "kube-api-access-6qjz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:40.675940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.675895    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qjz9\" (UniqueName: \"kubernetes.io/projected/46aed14a-2163-408d-bb68-fa6951479193-kube-api-access-6qjz9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:40.697152 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:40.697100    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aed14a-2163-408d-bb68-fa6951479193-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:41.085947 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.085910    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"
Oct 31 02:13:41.086521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.085912    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp" event={"ID":"9cdf2c36-fc81-44e6-b79b-4b6ccd770189","Type":"ContainerDied","Data":"f9a2e421048642f36a0f7c0dfa569616ea025d2805aadadd6b121ccccc5c6ce5"}
Oct 31 02:13:41.086521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.086096    5293 scope.go:122] "RemoveContainer" containerID="2cb1a8a685caea5438f733adf141ef4c3db961a4c487f1489d4ef2208afb7bd5"
Oct 31 02:13:41.087696 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.087675    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"
Oct 31 02:13:41.087862 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.087682    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t" event={"ID":"46aed14a-2163-408d-bb68-fa6951479193","Type":"ContainerDied","Data":"33abd207c44788dc7f8eac9a2d09310495adcf2349bc521d32ef1bb561fed9a3"}
Oct 31 02:13:41.088005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.087982    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33abd207c44788dc7f8eac9a2d09310495adcf2349bc521d32ef1bb561fed9a3"
Oct 31 02:13:41.090290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.090254    5293 generic.go:356] "Generic (PLEG): container finished" podID="0f264f44-9b2a-405f-bb7a-62e769613363" containerID="15667c201cc0c23d5f7857f61a4ffff8c5e2ff3ba2a010bdb39b762f22d0ec45" exitCode=0
Oct 31 02:13:41.090405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.090382    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx" event={"ID":"0f264f44-9b2a-405f-bb7a-62e769613363","Type":"ContainerDied","Data":"15667c201cc0c23d5f7857f61a4ffff8c5e2ff3ba2a010bdb39b762f22d0ec45"}
Oct 31 02:13:41.097586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.097561    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66" event={"ID":"87139ff5-903f-424b-9e99-b8740dc075e2","Type":"ContainerDied","Data":"844c1e49f16c072f90fff368278e4e81d0e435ccf06ee6a68ffae91b9c6c707d"}
Oct 31 02:13:41.097708 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.097659    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"
Oct 31 02:13:41.147270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.143906    5293 scope.go:122] "RemoveContainer" containerID="2142d3f0f394839a8e618233859f769c0df786f17a8f39454cd315747b2b8966"
Oct 31 02:13:41.159923 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.159890    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"]
Oct 31 02:13:41.164742 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.164711    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-zthqp"]
Oct 31 02:13:41.175205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.175169    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"]
Oct 31 02:13:41.179286 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:41.179261    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-xwm66"]
Oct 31 02:13:41.783173 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:41.783133    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:13:42.181754 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.181727    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" path="/var/lib/kubelet/pods/87139ff5-903f-424b-9e99-b8740dc075e2/volumes"
Oct 31 02:13:42.182631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.182604    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" path="/var/lib/kubelet/pods/9cdf2c36-fc81-44e6-b79b-4b6ccd770189/volumes"
Oct 31 02:13:42.360395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.358327    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"]
Oct 31 02:13:42.360395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.358531    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy" containerID="cri-o://2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da" gracePeriod=30
Oct 31 02:13:42.403200 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.403174    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:42.538420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.538321    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f264f44-9b2a-405f-bb7a-62e769613363-kube-api-access-6cvnk\" (UniqueName: \"kubernetes.io/projected/0f264f44-9b2a-405f-bb7a-62e769613363-kube-api-access-6cvnk\") pod \"0f264f44-9b2a-405f-bb7a-62e769613363\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") "
Oct 31 02:13:42.538605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.538456    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-openshift-ca\") pod \"0f264f44-9b2a-405f-bb7a-62e769613363\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") "
Oct 31 02:13:42.538858 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.538797    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-openshift-ca" pod "0f264f44-9b2a-405f-bb7a-62e769613363" (UID: "0f264f44-9b2a-405f-bb7a-62e769613363"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:42.538981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.538935    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-xks-ca\") pod \"0f264f44-9b2a-405f-bb7a-62e769613363\" (UID: \"0f264f44-9b2a-405f-bb7a-62e769613363\") "
Oct 31 02:13:42.540444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.540416    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:42.548877 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.548843    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-xks-ca" pod "0f264f44-9b2a-405f-bb7a-62e769613363" (UID: "0f264f44-9b2a-405f-bb7a-62e769613363"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:42.549016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.548888    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f264f44-9b2a-405f-bb7a-62e769613363-kube-api-access-6cvnk" pod "0f264f44-9b2a-405f-bb7a-62e769613363" (UID: "0f264f44-9b2a-405f-bb7a-62e769613363"). InnerVolumeSpecName "kube-api-access-6cvnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:42.642552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.642519    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cvnk\" (UniqueName: \"kubernetes.io/projected/0f264f44-9b2a-405f-bb7a-62e769613363-kube-api-access-6cvnk\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:42.642552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:42.642555    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f264f44-9b2a-405f-bb7a-62e769613363-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:43.180690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.180652    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx" event={"ID":"0f264f44-9b2a-405f-bb7a-62e769613363","Type":"ContainerDied","Data":"e42c46ef969f34080a91da146e1fcd1be1f1f714d84a91c0bea5d9bcfd4be837"}
Oct 31 02:13:43.180690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.180674    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"
Oct 31 02:13:43.180879 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.180703    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42c46ef969f34080a91da146e1fcd1be1f1f714d84a91c0bea5d9bcfd4be837"
Oct 31 02:13:43.780662 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.780629    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"]
Oct 31 02:13:43.798945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.798913    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"]
Oct 31 02:13:43.813103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.813071    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"]
Oct 31 02:13:43.834011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.825410    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"]
Oct 31 02:13:43.834011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.825747    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy" containerID="cri-o://462f9ce23423650f45ed18d38fcdf3eb134d24c0d3d6df35bbf38c04c9ea8c66" gracePeriod=30
Oct 31 02:13:43.846752 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.846707    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"]
Oct 31 02:13:43.847261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:43.847176    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy" containerID="cri-o://1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68" gracePeriod=30
Oct 31 02:13:46.832728 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.832687    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"]
Oct 31 02:13:46.833948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.833925    5293 cpu_manager.go:414] "containerMap: removing container" podUID="46aed14a-2163-408d-bb68-fa6951479193" containerName="revoke-keys"
Oct 31 02:13:46.834008 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.833951    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46aed14a-2163-408d-bb68-fa6951479193" containerName="revoke-keys"
Oct 31 02:13:46.834008 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.833967    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy"
Oct 31 02:13:46.834008 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.833974    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy"
Oct 31 02:13:46.834008 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.834001    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0f264f44-9b2a-405f-bb7a-62e769613363" containerName="revoke-keys"
Oct 31 02:13:46.834163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.834009    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f264f44-9b2a-405f-bb7a-62e769613363" containerName="revoke-keys"
Oct 31 02:13:46.834163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.834092    5293 cpu_manager.go:414] "containerMap: removing container" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy"
Oct 31 02:13:46.834163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.834100    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy"
Oct 31 02:13:46.834277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.834263    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="46aed14a-2163-408d-bb68-fa6951479193" containerName="revoke-keys"
Oct 31 02:13:46.834313 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.834282    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdf2c36-fc81-44e6-b79b-4b6ccd770189" containerName="istio-proxy"
Oct 31 02:13:46.834313 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.834296    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f264f44-9b2a-405f-bb7a-62e769613363" containerName="revoke-keys"
Oct 31 02:13:46.834380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.834315    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="87139ff5-903f-424b-9e99-b8740dc075e2" containerName="istio-proxy"
Oct 31 02:13:46.838622 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.838597    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:46.842197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.842173    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:13:46.842328 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.842239    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:13:46.850536 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.850008    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"]
Oct 31 02:13:46.856512 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.853053    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ftq\" (UniqueName: \"kubernetes.io/projected/f7920f04-647f-4a86-a616-c0bebac0a084-kube-api-access-z7ftq\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:46.856512 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.853393    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:46.856512 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.853631    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:46.881274 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.881239    5293 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:13:46.881426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.881312    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:13:46.998065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.992023    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:46.998065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.992287    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:46.998065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:46.992393    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ftq\" (UniqueName: \"kubernetes.io/projected/f7920f04-647f-4a86-a616-c0bebac0a084-kube-api-access-z7ftq\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:47.015960 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.015902    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:47.041871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.041838    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ftq\" (UniqueName: \"kubernetes.io/projected/f7920f04-647f-4a86-a616-c0bebac0a084-kube-api-access-z7ftq\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:47.042779 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.042287    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c463dc5-7gxmm\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:47.166872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.166843    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:47.288525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.288495    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:13:47.468266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.468213    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"]
Oct 31 02:13:47.766324 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.766295    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 02:13:47.864342 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864302    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-kube-api-access-wmpj9\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-kube-api-access-wmpj9\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.864856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864422    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-podinfo\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.864856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864543    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-data\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.864856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864656    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-token\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-token\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.864856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864701    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-credential-socket\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.864856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864753    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-socket\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.865197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864895    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aefe17a5-e46f-424f-82bf-b030fa88e212-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefe17a5-e46f-424f-82bf-b030fa88e212-istiod-ca-cert\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.865197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864955    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-certs\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.865197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.864989    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-envoy\") pod \"aefe17a5-e46f-424f-82bf-b030fa88e212\" (UID: \"aefe17a5-e46f-424f-82bf-b030fa88e212\") "
Oct 31 02:13:47.866409 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.866377    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-credential-socket" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:47.867091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.867054    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-certs" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:47.867413 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.867389    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-socket" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:47.867544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.867518    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-kube-api-access-wmpj9" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "kube-api-access-wmpj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:47.867606 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.867547    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefe17a5-e46f-424f-82bf-b030fa88e212-istiod-ca-cert" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:47.867822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.867802    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-data" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:47.871307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.871247    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-envoy" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:47.871718 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.871685    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-podinfo" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:47.872628 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.872604    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-token" pod "aefe17a5-e46f-424f-82bf-b030fa88e212" (UID: "aefe17a5-e46f-424f-82bf-b030fa88e212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:47.967148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967117    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:47.967148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967146    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:47.967148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967156    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:47.967401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967164    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefe17a5-e46f-424f-82bf-b030fa88e212-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:47.967401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967172    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:47.967401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967183    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:47.967401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967191    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpj9\" (UniqueName: \"kubernetes.io/projected/aefe17a5-e46f-424f-82bf-b030fa88e212-kube-api-access-wmpj9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:47.967401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967200    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:47.967401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:47.967207    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefe17a5-e46f-424f-82bf-b030fa88e212-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:48.283323 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.283229    5293 generic.go:356] "Generic (PLEG): container finished" podID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerID="2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da" exitCode=0
Oct 31 02:13:48.283506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.283347    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"
Oct 31 02:13:48.283506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.283371    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" event={"ID":"aefe17a5-e46f-424f-82bf-b030fa88e212","Type":"ContainerDied","Data":"2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da"}
Oct 31 02:13:48.283506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.283437    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh" event={"ID":"aefe17a5-e46f-424f-82bf-b030fa88e212","Type":"ContainerDied","Data":"c4862286abd3e280993b439507d6a98c601c07d4f63264bb506df89d3fdae057"}
Oct 31 02:13:48.283506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.283465    5293 scope.go:122] "RemoveContainer" containerID="2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da"
Oct 31 02:13:48.288500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.287331    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm" event={"ID":"f7920f04-647f-4a86-a616-c0bebac0a084","Type":"ContainerStarted","Data":"9dd82c039820d0f4e5f1bd6e5893e526a53085d27d46f662eba0a436cc49c044"}
Oct 31 02:13:48.288500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.287388    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm" event={"ID":"f7920f04-647f-4a86-a616-c0bebac0a084","Type":"ContainerStarted","Data":"5142d08366f9055b2676e0995c677263cb81117033644b2beda3ade706d0f8bb"}
Oct 31 02:13:48.297147 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.297127    5293 scope.go:122] "RemoveContainer" containerID="2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da"
Oct 31 02:13:48.297521 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:48.297499    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da\": container with ID starting with 2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da not found: ID does not exist" containerID="2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da"
Oct 31 02:13:48.297627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.297528    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da"} err="failed to get container status \"2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da\": rpc error: code = NotFound desc = could not find container \"2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da\": container with ID starting with 2f377bd65b5e54c0c50e49d1f4f97e75b1fd910c46881b2adf61de48d50353da not found: ID does not exist"
Oct 31 02:13:48.303942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.303909    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"]
Oct 31 02:13:48.317492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.317470    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-dbfsh"]
Oct 31 02:13:48.328912 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.328857    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm" podStartSLOduration=2.328693692 podStartE2EDuration="2.328693692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:48.32535093 +0000 UTC m=+3186.273999326" watchObservedRunningTime="2026-10-31 02:13:48.328693692 +0000 UTC m=+3186.277342083"
Oct 31 02:13:49.023067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:48.995451    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:13:49.548732 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.546776    5293 generic.go:356] "Generic (PLEG): container finished" podID="f7920f04-647f-4a86-a616-c0bebac0a084" containerID="9dd82c039820d0f4e5f1bd6e5893e526a53085d27d46f662eba0a436cc49c044" exitCode=0
Oct 31 02:13:49.548732 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.546951    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm" event={"ID":"f7920f04-647f-4a86-a616-c0bebac0a084","Type":"ContainerDied","Data":"9dd82c039820d0f4e5f1bd6e5893e526a53085d27d46f662eba0a436cc49c044"}
Oct 31 02:13:49.549736 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.549693    5293 generic.go:356] "Generic (PLEG): container finished" podID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerID="462f9ce23423650f45ed18d38fcdf3eb134d24c0d3d6df35bbf38c04c9ea8c66" exitCode=0
Oct 31 02:13:49.550146 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.549943    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" event={"ID":"d167f5c4-6e60-4517-8904-450dd8e8416c","Type":"ContainerDied","Data":"462f9ce23423650f45ed18d38fcdf3eb134d24c0d3d6df35bbf38c04c9ea8c66"}
Oct 31 02:13:49.674867 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.674838    5293 scope.go:122] "RemoveContainer" containerID="0edc729b303871b338800bedaef2a8beff3af3ce7a9cbeb3ab4c3e736ee24c3c"
Oct 31 02:13:49.691992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.691939    5293 scope.go:122] "RemoveContainer" containerID="66b0d10781ec47e00e632e37e7a1e0da5286847669010bb7fb67fbd38ef81f93"
Oct 31 02:13:49.707157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.707135    5293 scope.go:122] "RemoveContainer" containerID="8e9ad5f3512c5a0095855aab51d3eb76706b01b8b5b8928e58464b8405bb7a9d"
Oct 31 02:13:49.730540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.729716    5293 scope.go:122] "RemoveContainer" containerID="4946f390a5ce951ba04d07175dfa2cfab4a873c60262d3967a17e5fb3a718f6a"
Oct 31 02:13:49.750868 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.750847    5293 scope.go:122] "RemoveContainer" containerID="ec589c41c21fcb936f4da802d5535f621c117bb550c14e3e68a8526b59bb7e38"
Oct 31 02:13:49.815281 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.815258    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:13:49.819766 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.819686    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 02:13:49.959081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.919359    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-data\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:49.967161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.967120    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-istio-token\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-istio-token\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:49.967447 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.967427    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-credential-socket\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:49.967568 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.967554    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293991cd-eab8-48f3-882f-bf1dc275809d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293991cd-eab8-48f3-882f-bf1dc275809d-istiod-ca-cert\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:49.967689 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.967675    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-certs\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:49.967799 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.967785    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-envoy\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:49.967918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.967904    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-kube-api-access-5h7xq\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-kube-api-access-5h7xq\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:49.968076 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.968061    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-socket\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:49.993169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.970300    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-data\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:50.006464 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.006439    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-podinfo\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:50.042185 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.042148    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-socket\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051165    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-credential-socket\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051214    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-token\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-token\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051257    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-kube-api-access-j5jqp\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-kube-api-access-j5jqp\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051393    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/293991cd-eab8-48f3-882f-bf1dc275809d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293991cd-eab8-48f3-882f-bf1dc275809d-istio-podinfo\") pod \"293991cd-eab8-48f3-882f-bf1dc275809d\" (UID: \"293991cd-eab8-48f3-882f-bf1dc275809d\") "
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.941203    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-data" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.971741    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293991cd-eab8-48f3-882f-bf1dc275809d-istiod-ca-cert" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.977275    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-data" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.977449    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-certs" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:49.977929    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-credential-socket" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.008131    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-socket" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.031315    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-kube-api-access-5h7xq" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "kube-api-access-5h7xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.032093    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-istio-token" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.050973    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-envoy" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051240    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-socket" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051417    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-credential-socket" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051424    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-envoy\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051594    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d167f5c4-6e60-4517-8904-450dd8e8416c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167f5c4-6e60-4517-8904-450dd8e8416c-istiod-ca-cert\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.051632    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-certs\") pod \"d167f5c4-6e60-4517-8904-450dd8e8416c\" (UID: \"d167f5c4-6e60-4517-8904-450dd8e8416c\") "
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052711    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052733    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052751    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052764    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052777    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052789    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052804    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293991cd-eab8-48f3-882f-bf1dc275809d-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052817    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052829    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293991cd-eab8-48f3-882f-bf1dc275809d-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052844    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7xq\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-kube-api-access-5h7xq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.052859    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.053106    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-certs" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.054421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.053423    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d167f5c4-6e60-4517-8904-450dd8e8416c-istiod-ca-cert" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:50.083742 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.071166    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-envoy" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.113066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.097390    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:13:50.177068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.176801    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-podinfo" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:50.224173 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.217637    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.224173 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.217672    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.224173 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.217688    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167f5c4-6e60-4517-8904-450dd8e8416c-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.224173 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.217704    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167f5c4-6e60-4517-8904-450dd8e8416c-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.255313 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.255259    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/293991cd-eab8-48f3-882f-bf1dc275809d-istio-podinfo" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:50.313460 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.313429    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-kube-api-access-j5jqp" pod "293991cd-eab8-48f3-882f-bf1dc275809d" (UID: "293991cd-eab8-48f3-882f-bf1dc275809d"). InnerVolumeSpecName "kube-api-access-j5jqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:50.320141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.320113    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jqp\" (UniqueName: \"kubernetes.io/projected/293991cd-eab8-48f3-882f-bf1dc275809d-kube-api-access-j5jqp\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.320141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.320145    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293991cd-eab8-48f3-882f-bf1dc275809d-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.320611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.320585    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" path="/var/lib/kubelet/pods/aefe17a5-e46f-424f-82bf-b030fa88e212/volumes"
Oct 31 02:13:50.323231 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.323207    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-token" pod "d167f5c4-6e60-4517-8904-450dd8e8416c" (UID: "d167f5c4-6e60-4517-8904-450dd8e8416c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:50.323705 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.323685    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"]
Oct 31 02:13:50.421297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.421272    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d167f5c4-6e60-4517-8904-450dd8e8416c-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.560582 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.560551    5293 generic.go:356] "Generic (PLEG): container finished" podID="293991cd-eab8-48f3-882f-bf1dc275809d" containerID="1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68" exitCode=0
Oct 31 02:13:50.560782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.560599    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"
Oct 31 02:13:50.560782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.560633    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" event={"ID":"293991cd-eab8-48f3-882f-bf1dc275809d","Type":"ContainerDied","Data":"1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68"}
Oct 31 02:13:50.560782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.560682    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq" event={"ID":"293991cd-eab8-48f3-882f-bf1dc275809d","Type":"ContainerDied","Data":"0df7871af3eef04f8d41d962f9cc610fd887907511a6437425bef9f9d1774bd9"}
Oct 31 02:13:50.560782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.560711    5293 scope.go:122] "RemoveContainer" containerID="1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68"
Oct 31 02:13:50.564872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.562541    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"
Oct 31 02:13:50.564872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.562583    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z" event={"ID":"d167f5c4-6e60-4517-8904-450dd8e8416c","Type":"ContainerDied","Data":"a44d95799b29669596ce4e4ac609e8dfef0cb84623afef918246db89c0156d84"}
Oct 31 02:13:50.648584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.638873    5293 scope.go:122] "RemoveContainer" containerID="1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68"
Oct 31 02:13:50.656264 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:50.656225    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68\": container with ID starting with 1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68 not found: ID does not exist" containerID="1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68"
Oct 31 02:13:50.656444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.656276    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68"} err="failed to get container status \"1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68\": rpc error: code = NotFound desc = could not find container \"1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68\": container with ID starting with 1424916f0e8181caa6f152975b7617ebabf1aa843ab71fa7934c35c9616e6f68 not found: ID does not exist"
Oct 31 02:13:50.656444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.656303    5293 scope.go:122] "RemoveContainer" containerID="462f9ce23423650f45ed18d38fcdf3eb134d24c0d3d6df35bbf38c04c9ea8c66"
Oct 31 02:13:50.674333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.674272    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"]
Oct 31 02:13:50.683851 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.683575    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-b488z"]
Oct 31 02:13:50.686479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.686454    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"]
Oct 31 02:13:50.690754 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.690734    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-4kvhq"]
Oct 31 02:13:50.853636 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.853611    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:50.945083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.945016    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-xks-ca\") pod \"f7920f04-647f-4a86-a616-c0bebac0a084\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") "
Oct 31 02:13:50.945251 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.945148    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-openshift-ca\") pod \"f7920f04-647f-4a86-a616-c0bebac0a084\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") "
Oct 31 02:13:50.945539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.945512    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-openshift-ca" pod "f7920f04-647f-4a86-a616-c0bebac0a084" (UID: "f7920f04-647f-4a86-a616-c0bebac0a084"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:50.945539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.945518    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7920f04-647f-4a86-a616-c0bebac0a084-kube-api-access-z7ftq\" (UniqueName: \"kubernetes.io/projected/f7920f04-647f-4a86-a616-c0bebac0a084-kube-api-access-z7ftq\") pod \"f7920f04-647f-4a86-a616-c0bebac0a084\" (UID: \"f7920f04-647f-4a86-a616-c0bebac0a084\") "
Oct 31 02:13:50.946292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.946270    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.947424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.947400    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-xks-ca" pod "f7920f04-647f-4a86-a616-c0bebac0a084" (UID: "f7920f04-647f-4a86-a616-c0bebac0a084"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:50.948013 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:50.947991    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7920f04-647f-4a86-a616-c0bebac0a084-kube-api-access-z7ftq" pod "f7920f04-647f-4a86-a616-c0bebac0a084" (UID: "f7920f04-647f-4a86-a616-c0bebac0a084"). InnerVolumeSpecName "kube-api-access-z7ftq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:51.073167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:51.064509    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7920f04-647f-4a86-a616-c0bebac0a084-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:51.084066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:51.073764    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ftq\" (UniqueName: \"kubernetes.io/projected/f7920f04-647f-4a86-a616-c0bebac0a084-kube-api-access-z7ftq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:51.630424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:51.630290    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm" event={"ID":"f7920f04-647f-4a86-a616-c0bebac0a084","Type":"ContainerDied","Data":"5142d08366f9055b2676e0995c677263cb81117033644b2beda3ade706d0f8bb"}
Oct 31 02:13:51.630935 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:51.630915    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5142d08366f9055b2676e0995c677263cb81117033644b2beda3ade706d0f8bb"
Oct 31 02:13:51.631190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:51.631172    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"
Oct 31 02:13:52.183393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:52.183363    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" path="/var/lib/kubelet/pods/293991cd-eab8-48f3-882f-bf1dc275809d/volumes"
Oct 31 02:13:52.183941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:52.183925    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" path="/var/lib/kubelet/pods/d167f5c4-6e60-4517-8904-450dd8e8416c/volumes"
Oct 31 02:13:53.282416 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:53.282377    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"]
Oct 31 02:13:53.282915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:53.282590    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy" containerID="cri-o://6f8a637ae81c129e39490843a1575424a7fea8eea6785e083782135fc3d5ed84" gracePeriod=30
Oct 31 02:13:55.324348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.324307    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69" podUID="2092a552-680f-4bd0-9688-b7b0defe302d" containerName="payload-processing" containerID="cri-o://d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c" gracePeriod=25
Oct 31 02:13:55.620219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.620190    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:13:55.747743 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.747712    5293 generic.go:356] "Generic (PLEG): container finished" podID="2092a552-680f-4bd0-9688-b7b0defe302d" containerID="d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c" exitCode=0
Oct 31 02:13:55.747952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.747778    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"
Oct 31 02:13:55.747952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.747801    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69" event={"ID":"2092a552-680f-4bd0-9688-b7b0defe302d","Type":"ContainerDied","Data":"d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c"}
Oct 31 02:13:55.747952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.747860    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69" event={"ID":"2092a552-680f-4bd0-9688-b7b0defe302d","Type":"ContainerDied","Data":"47088fd3e6fffc35596181f025d0c18f798fa73e1e48cdb58aa85697379f6498"}
Oct 31 02:13:55.747952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.747884    5293 scope.go:122] "RemoveContainer" containerID="d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c"
Oct 31 02:13:55.771497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.770257    5293 scope.go:122] "RemoveContainer" containerID="d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c"
Oct 31 02:13:55.771497 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:13:55.770717    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c\": container with ID starting with d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c not found: ID does not exist" containerID="d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c"
Oct 31 02:13:55.771497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.770753    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c"} err="failed to get container status \"d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c\": rpc error: code = NotFound desc = could not find container \"d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c\": container with ID starting with d14909318e10a9417e16aa931458862852a9e3b743309cfc171dc6a2926a305c not found: ID does not exist"
Oct 31 02:13:55.771497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.771353    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-routing-overlay-volume\") pod \"2092a552-680f-4bd0-9688-b7b0defe302d\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") "
Oct 31 02:13:55.771874 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.771770    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-provider-credentials\") pod \"2092a552-680f-4bd0-9688-b7b0defe302d\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") "
Oct 31 02:13:55.771874 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.771806    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-plugins-config-volume\") pod \"2092a552-680f-4bd0-9688-b7b0defe302d\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") "
Oct 31 02:13:55.771984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.771875    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-kube-api-access-xjsjn\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-kube-api-access-xjsjn\") pod \"2092a552-680f-4bd0-9688-b7b0defe302d\" (UID: \"2092a552-680f-4bd0-9688-b7b0defe302d\") "
Oct 31 02:13:55.785070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.773681    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-plugins-config-volume" pod "2092a552-680f-4bd0-9688-b7b0defe302d" (UID: "2092a552-680f-4bd0-9688-b7b0defe302d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:55.785070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.774164    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-routing-overlay-volume" pod "2092a552-680f-4bd0-9688-b7b0defe302d" (UID: "2092a552-680f-4bd0-9688-b7b0defe302d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:55.785070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.775397    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:55.785070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.776462    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2092a552-680f-4bd0-9688-b7b0defe302d-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:55.835653 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.827248    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-provider-credentials" pod "2092a552-680f-4bd0-9688-b7b0defe302d" (UID: "2092a552-680f-4bd0-9688-b7b0defe302d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:55.836170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.836019    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-kube-api-access-xjsjn" pod "2092a552-680f-4bd0-9688-b7b0defe302d" (UID: "2092a552-680f-4bd0-9688-b7b0defe302d"). InnerVolumeSpecName "kube-api-access-xjsjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:55.878593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.878561    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:55.878593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:55.878591    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjsjn\" (UniqueName: \"kubernetes.io/projected/2092a552-680f-4bd0-9688-b7b0defe302d-kube-api-access-xjsjn\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:56.072715 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:56.072600    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"]
Oct 31 02:13:56.076988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:56.076958    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-56ddmhs69"]
Oct 31 02:13:56.179948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:56.179906    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2092a552-680f-4bd0-9688-b7b0defe302d" path="/var/lib/kubelet/pods/2092a552-680f-4bd0-9688-b7b0defe302d/volumes"
Oct 31 02:13:58.207065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:58.205684    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"]
Oct 31 02:13:58.850132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:58.842367    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"]
Oct 31 02:13:58.880069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:58.876850    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" containerID="cri-o://a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb" gracePeriod=45
Oct 31 02:13:58.960712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:58.947803    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" containerID="cri-o://2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a" gracePeriod=45
Oct 31 02:13:58.960712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:58.956800    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" containerID="cri-o://60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a" gracePeriod=45
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.311189    5293 generic.go:356] "Generic (PLEG): container finished" podID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerID="6f8a637ae81c129e39490843a1575424a7fea8eea6785e083782135fc3d5ed84" exitCode=0
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.311355    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" event={"ID":"10df5a77-8db8-41cb-866d-494e7832c0d1","Type":"ContainerDied","Data":"6f8a637ae81c129e39490843a1575424a7fea8eea6785e083782135fc3d5ed84"}
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.311844    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": dial tcp 10.133.0.77:8000: connect: connection refused"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.329278    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"]
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330509    5293 cpu_manager.go:414] "containerMap: removing container" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330531    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330558    5293 cpu_manager.go:414] "containerMap: removing container" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330567    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330603    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330613    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330641    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2092a552-680f-4bd0-9688-b7b0defe302d" containerName="payload-processing"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330650    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2092a552-680f-4bd0-9688-b7b0defe302d" containerName="payload-processing"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330674    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f7920f04-647f-4a86-a616-c0bebac0a084" containerName="revoke-keys"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330681    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7920f04-647f-4a86-a616-c0bebac0a084" containerName="revoke-keys"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330916    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2092a552-680f-4bd0-9688-b7b0defe302d" containerName="payload-processing"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330935    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7920f04-647f-4a86-a616-c0bebac0a084" containerName="revoke-keys"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330953    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="293991cd-eab8-48f3-882f-bf1dc275809d" containerName="istio-proxy"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330964    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefe17a5-e46f-424f-82bf-b030fa88e212" containerName="istio-proxy"
Oct 31 02:13:59.344918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.330980    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d167f5c4-6e60-4517-8904-450dd8e8416c" containerName="istio-proxy"
Oct 31 02:13:59.376719 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.376146    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 31 02:13:59.384910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.381302    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": EOF"
Oct 31 02:13:59.386723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.386295    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"]
Oct 31 02:13:59.386723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.386440    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.391086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.390353    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:13:59.391086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.390468    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:13:59.413334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.413290    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.413913 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.413886    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492s2\" (UniqueName: \"kubernetes.io/projected/8428ea83-43a2-4f98-a98c-2ac754528306-kube-api-access-492s2\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.414376 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.414353    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.518397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.517473    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.518397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.517653    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.518397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.517763    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492s2\" (UniqueName: \"kubernetes.io/projected/8428ea83-43a2-4f98-a98c-2ac754528306-kube-api-access-492s2\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.518878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.518802    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.528517 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.528458    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.535866 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.535808    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-492s2\" (UniqueName: \"kubernetes.io/projected/8428ea83-43a2-4f98-a98c-2ac754528306-kube-api-access-492s2\") pod \"maas-api-revoke-sub-4681fa31-vdk8b\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.672367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.672158    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:13:59.680001 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.679968    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:13:59.778772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.778702    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:13:59.834880 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.834840    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-envoy\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.835260 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.835219    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10df5a77-8db8-41cb-866d-494e7832c0d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df5a77-8db8-41cb-866d-494e7832c0d1-istiod-ca-cert\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.835492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.835471    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-kube-api-access-xmq2n\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-kube-api-access-xmq2n\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.835660 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.835639    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-token\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-token\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.835891 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.835870    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-certs\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.836509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.836488    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-data\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.836701 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.836681    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-credential-socket\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.836883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.836854    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-podinfo\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.837034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.837016    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-socket\") pod \"10df5a77-8db8-41cb-866d-494e7832c0d1\" (UID: \"10df5a77-8db8-41cb-866d-494e7832c0d1\") "
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.836895    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-data" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.837115    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-certs" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.837468    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-credential-socket" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.837692    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10df5a77-8db8-41cb-866d-494e7832c0d1-istiod-ca-cert" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.838153    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-socket" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.838558    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.838574    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.838584    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.838594    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df5a77-8db8-41cb-866d-494e7832c0d1-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.839362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.838606    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.840965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.840518    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:13:59.844204 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.843266    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-kube-api-access-xmq2n" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "kube-api-access-xmq2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:59.844204 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.843388    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-envoy" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.844204 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.843770    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-token" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:59.844975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.844943    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-podinfo" pod "10df5a77-8db8-41cb-866d-494e7832c0d1" (UID: "10df5a77-8db8-41cb-866d-494e7832c0d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:59.869180 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.869151    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:13:59.939412 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.939345    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-home\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-home\") pod \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") "
Oct 31 02:13:59.939606 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.939461    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tls-certs\") pod \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") "
Oct 31 02:13:59.939691 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.939639    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-dshm\") pod \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") "
Oct 31 02:13:59.940136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.939796    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-dshm\") pod \"b4cc03be-968e-4464-9710-4d082d728072\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") "
Oct 31 02:13:59.940328 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940299    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-model-cache\") pod \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940344    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tmp-dir\") pod \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940390    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cc03be-968e-4464-9710-4d082d728072-kube-api-access-fpxwd\" (UniqueName: \"kubernetes.io/projected/b4cc03be-968e-4464-9710-4d082d728072-kube-api-access-fpxwd\") pod \"b4cc03be-968e-4464-9710-4d082d728072\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940504    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-model-cache\") pod \"b4cc03be-968e-4464-9710-4d082d728072\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940536    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef71ae1-dac7-470a-941d-e3c2f83f6940-kube-api-access-6h82b\" (UniqueName: \"kubernetes.io/projected/2ef71ae1-dac7-470a-941d-e3c2f83f6940-kube-api-access-6h82b\") pod \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\" (UID: \"2ef71ae1-dac7-470a-941d-e3c2f83f6940\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940573    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-kube-api-access-g2fkr\" (UniqueName: \"kubernetes.io/projected/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-kube-api-access-g2fkr\") pod \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940638    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-tmp-dir\") pod \"b4cc03be-968e-4464-9710-4d082d728072\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940669    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-model-cache" pod "2ef71ae1-dac7-470a-941d-e3c2f83f6940" (UID: "2ef71ae1-dac7-470a-941d-e3c2f83f6940"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940761    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cc03be-968e-4464-9710-4d082d728072-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cc03be-968e-4464-9710-4d082d728072-tls-certs\") pod \"b4cc03be-968e-4464-9710-4d082d728072\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940848    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tmp-dir\") pod \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") "
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940878    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-tmp-dir" pod "b4cc03be-968e-4464-9710-4d082d728072" (UID: "b4cc03be-968e-4464-9710-4d082d728072"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.940900    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tmp-dir" pod "2ef71ae1-dac7-470a-941d-e3c2f83f6940" (UID: "2ef71ae1-dac7-470a-941d-e3c2f83f6940"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941254    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-model-cache" pod "b4cc03be-968e-4464-9710-4d082d728072" (UID: "b4cc03be-968e-4464-9710-4d082d728072"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941486    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tmp-dir" pod "6817381f-1ab2-4b8b-b580-d3b37ee07f6d" (UID: "6817381f-1ab2-4b8b-b580-d3b37ee07f6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.941813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941808    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941834    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmq2n\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-kube-api-access-xmq2n\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941852    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941866    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941880    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941895    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941910    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941925    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df5a77-8db8-41cb-866d-494e7832c0d1-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.941938    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.942604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.942268    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-home" pod "2ef71ae1-dac7-470a-941d-e3c2f83f6940" (UID: "2ef71ae1-dac7-470a-941d-e3c2f83f6940"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.948636 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.948495    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef71ae1-dac7-470a-941d-e3c2f83f6940-kube-api-access-6h82b" pod "2ef71ae1-dac7-470a-941d-e3c2f83f6940" (UID: "2ef71ae1-dac7-470a-941d-e3c2f83f6940"). InnerVolumeSpecName "kube-api-access-6h82b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:59.948636 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.948597    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-dshm" pod "2ef71ae1-dac7-470a-941d-e3c2f83f6940" (UID: "2ef71ae1-dac7-470a-941d-e3c2f83f6940"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.948819 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.948628    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-kube-api-access-g2fkr" pod "6817381f-1ab2-4b8b-b580-d3b37ee07f6d" (UID: "6817381f-1ab2-4b8b-b580-d3b37ee07f6d"). InnerVolumeSpecName "kube-api-access-g2fkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:59.948884 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.948814    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tls-certs" pod "2ef71ae1-dac7-470a-941d-e3c2f83f6940" (UID: "2ef71ae1-dac7-470a-941d-e3c2f83f6940"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:59.949264 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.949240    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-dshm" pod "b4cc03be-968e-4464-9710-4d082d728072" (UID: "b4cc03be-968e-4464-9710-4d082d728072"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.950722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.950700    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cc03be-968e-4464-9710-4d082d728072-kube-api-access-fpxwd" pod "b4cc03be-968e-4464-9710-4d082d728072" (UID: "b4cc03be-968e-4464-9710-4d082d728072"). InnerVolumeSpecName "kube-api-access-fpxwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:59.951180 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:13:59.951162    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cc03be-968e-4464-9710-4d082d728072-tls-certs" pod "b4cc03be-968e-4464-9710-4d082d728072" (UID: "b4cc03be-968e-4464-9710-4d082d728072"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:00.019133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.019093    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"]
Oct 31 02:14:00.019511 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:14:00.019480    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8428ea83_43a2_4f98_a98c_2ac754528306.slice/crio-e613b9801e279491ce6cffafa5f9731b8845fc39ac0355f7e02aba98e9d5ca63 WatchSource:0}: Error finding container e613b9801e279491ce6cffafa5f9731b8845fc39ac0355f7e02aba98e9d5ca63: Status 404 returned error can't find the container with id e613b9801e279491ce6cffafa5f9731b8845fc39ac0355f7e02aba98e9d5ca63
Oct 31 02:14:00.042837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.042806    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tls-certs\") pod \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") "
Oct 31 02:14:00.042976 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.042863    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-home\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-home\") pod \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") "
Oct 31 02:14:00.042976 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.042958    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-home\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-home\") pod \"b4cc03be-968e-4464-9710-4d082d728072\" (UID: \"b4cc03be-968e-4464-9710-4d082d728072\") "
Oct 31 02:14:00.043262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.043245    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-model-cache\") pod \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") "
Oct 31 02:14:00.043350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.043283    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-dshm\") pod \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\" (UID: \"6817381f-1ab2-4b8b-b580-d3b37ee07f6d\") "
Oct 31 02:14:00.043421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.043352    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-home" pod "b4cc03be-968e-4464-9710-4d082d728072" (UID: "b4cc03be-968e-4464-9710-4d082d728072"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.043562    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-home" pod "6817381f-1ab2-4b8b-b580-d3b37ee07f6d" (UID: "6817381f-1ab2-4b8b-b580-d3b37ee07f6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.043620    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-model-cache" pod "6817381f-1ab2-4b8b-b580-d3b37ee07f6d" (UID: "6817381f-1ab2-4b8b-b580-d3b37ee07f6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044000    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044021    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpxwd\" (UniqueName: \"kubernetes.io/projected/b4cc03be-968e-4464-9710-4d082d728072-kube-api-access-fpxwd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044082    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h82b\" (UniqueName: \"kubernetes.io/projected/2ef71ae1-dac7-470a-941d-e3c2f83f6940-kube-api-access-6h82b\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044093    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2fkr\" (UniqueName: \"kubernetes.io/projected/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-kube-api-access-g2fkr\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044105    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cc03be-968e-4464-9710-4d082d728072-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044114    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044121    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044130    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef71ae1-dac7-470a-941d-e3c2f83f6940-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044138    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044146    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef71ae1-dac7-470a-941d-e3c2f83f6940-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.044598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.044156    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4cc03be-968e-4464-9710-4d082d728072-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.045993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.045966    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tls-certs" pod "6817381f-1ab2-4b8b-b580-d3b37ee07f6d" (UID: "6817381f-1ab2-4b8b-b580-d3b37ee07f6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:00.047149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.047117    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-dshm" pod "6817381f-1ab2-4b8b-b580-d3b37ee07f6d" (UID: "6817381f-1ab2-4b8b-b580-d3b37ee07f6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:00.145557 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.145524    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.145557 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.145553    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6817381f-1ab2-4b8b-b580-d3b37ee07f6d-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.321958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.321920    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"
Oct 31 02:14:00.321958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.321948    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx" event={"ID":"10df5a77-8db8-41cb-866d-494e7832c0d1","Type":"ContainerDied","Data":"34859db7a97f3e635ece9fce689fcfa41d9c4e24a0e40c86ed12c08b979c6382"}
Oct 31 02:14:00.322265 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.322054    5293 scope.go:122] "RemoveContainer" containerID="6f8a637ae81c129e39490843a1575424a7fea8eea6785e083782135fc3d5ed84"
Oct 31 02:14:00.323566 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.323545    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b" event={"ID":"8428ea83-43a2-4f98-a98c-2ac754528306","Type":"ContainerStarted","Data":"e613b9801e279491ce6cffafa5f9731b8845fc39ac0355f7e02aba98e9d5ca63"}
Oct 31 02:14:00.325657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.325631    5293 generic.go:356] "Generic (PLEG): container finished" podID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerID="2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a" exitCode=0
Oct 31 02:14:00.325785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.325701    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"
Oct 31 02:14:00.325888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.325866    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" event={"ID":"6817381f-1ab2-4b8b-b580-d3b37ee07f6d","Type":"ContainerDied","Data":"2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a"}
Oct 31 02:14:00.325952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.325907    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb" event={"ID":"6817381f-1ab2-4b8b-b580-d3b37ee07f6d","Type":"ContainerDied","Data":"3956ee9c10bbf64a8ace50ef650d6ede898773d492cf7d739144061eb1734763"}
Oct 31 02:14:00.327690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.327668    5293 generic.go:356] "Generic (PLEG): container finished" podID="b4cc03be-968e-4464-9710-4d082d728072" containerID="60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a" exitCode=0
Oct 31 02:14:00.327822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.327723    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"
Oct 31 02:14:00.327822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.327794    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" event={"ID":"b4cc03be-968e-4464-9710-4d082d728072","Type":"ContainerDied","Data":"60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a"}
Oct 31 02:14:00.327822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.327820    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw" event={"ID":"b4cc03be-968e-4464-9710-4d082d728072","Type":"ContainerDied","Data":"d01c2bc0fe8ead32385a7ff84f58818d961bed12ecaabb8d12f270c03e2f60fd"}
Oct 31 02:14:00.329830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.329800    5293 generic.go:356] "Generic (PLEG): container finished" podID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerID="a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb" exitCode=0
Oct 31 02:14:00.329932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.329854    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"
Oct 31 02:14:00.329986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.329926    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" event={"ID":"2ef71ae1-dac7-470a-941d-e3c2f83f6940","Type":"ContainerDied","Data":"a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb"}
Oct 31 02:14:00.329986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.329961    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf" event={"ID":"2ef71ae1-dac7-470a-941d-e3c2f83f6940","Type":"ContainerDied","Data":"c72c95c450566932c121db1261275e7dd6ca4d502664334e8d52a329db8c442a"}
Oct 31 02:14:00.340086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.340062    5293 scope.go:122] "RemoveContainer" containerID="2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a"
Oct 31 02:14:00.395488 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.395459    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"]
Oct 31 02:14:00.421109 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.417088    5293 scope.go:122] "RemoveContainer" containerID="2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a"
Oct 31 02:14:00.426166 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:14:00.425567    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a\": container with ID starting with 2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a not found: ID does not exist" containerID="2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a"
Oct 31 02:14:00.426166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.425624    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a"} err="failed to get container status \"2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a\": rpc error: code = NotFound desc = could not find container \"2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a\": container with ID starting with 2d491071f099929fb13e531685913d433b13f7db1b8648b1af87cb279d98fb9a not found: ID does not exist"
Oct 31 02:14:00.426166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.425660    5293 scope.go:122] "RemoveContainer" containerID="60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a"
Oct 31 02:14:00.458196 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.458099    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-rknbx"]
Oct 31 02:14:00.471843 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.471589    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"]
Oct 31 02:14:00.477997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.477657    5293 scope.go:122] "RemoveContainer" containerID="60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a"
Oct 31 02:14:00.478574 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:14:00.478273    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a\": container with ID starting with 60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a not found: ID does not exist" containerID="60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a"
Oct 31 02:14:00.478574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.478307    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a"} err="failed to get container status \"60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a\": rpc error: code = NotFound desc = could not find container \"60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a\": container with ID starting with 60d1d7cd3865130f25f36041bdd62eb11bff287f90c99e554f5ab6fdb774b19a not found: ID does not exist"
Oct 31 02:14:00.478574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.478335    5293 scope.go:122] "RemoveContainer" containerID="a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb"
Oct 31 02:14:00.479013 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.478976    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/simulator-w4-be6935-kserve-84d7b97987-pcxvb"]
Oct 31 02:14:00.483368 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.483325    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"]
Oct 31 02:14:00.488930 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.488901    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/premium-w4-be6935-kserve-7766745998-74lsw"]
Oct 31 02:14:00.498297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.496441    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"]
Oct 31 02:14:00.498297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.496798    5293 scope.go:122] "RemoveContainer" containerID="a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb"
Oct 31 02:14:00.498297 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:14:00.497974    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb\": container with ID starting with a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb not found: ID does not exist" containerID="a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb"
Oct 31 02:14:00.498297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.498009    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb"} err="failed to get container status \"a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb\": rpc error: code = NotFound desc = could not find container \"a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb\": container with ID starting with a354ea51eea4402e713f87262e94f4384503ef21b04ef1207f1efd36c6517cfb not found: ID does not exist"
Oct 31 02:14:00.499170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:00.499148    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-w4-be6935-kserve-8c7f9b865-rn7qf"]
Oct 31 02:14:01.341791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:01.341705    5293 generic.go:356] "Generic (PLEG): container finished" podID="8428ea83-43a2-4f98-a98c-2ac754528306" containerID="1a63e8828c6dbfd49dce5c77cc1160aec79a1d10696505e5a2964b83ab05411a" exitCode=0
Oct 31 02:14:01.342098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:01.341738    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b" event={"ID":"8428ea83-43a2-4f98-a98c-2ac754528306","Type":"ContainerDied","Data":"1a63e8828c6dbfd49dce5c77cc1160aec79a1d10696505e5a2964b83ab05411a"}
Oct 31 02:14:02.175585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.175545    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" path="/var/lib/kubelet/pods/10df5a77-8db8-41cb-866d-494e7832c0d1/volumes"
Oct 31 02:14:02.176159 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.176140    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" path="/var/lib/kubelet/pods/2ef71ae1-dac7-470a-941d-e3c2f83f6940/volumes"
Oct 31 02:14:02.176603 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.176589    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" path="/var/lib/kubelet/pods/6817381f-1ab2-4b8b-b580-d3b37ee07f6d/volumes"
Oct 31 02:14:02.177372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.177355    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cc03be-968e-4464-9710-4d082d728072" path="/var/lib/kubelet/pods/b4cc03be-968e-4464-9710-4d082d728072/volumes"
Oct 31 02:14:02.593356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.593330    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:14:02.723427 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.723395    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8428ea83-43a2-4f98-a98c-2ac754528306-kube-api-access-492s2\" (UniqueName: \"kubernetes.io/projected/8428ea83-43a2-4f98-a98c-2ac754528306-kube-api-access-492s2\") pod \"8428ea83-43a2-4f98-a98c-2ac754528306\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") "
Oct 31 02:14:02.723610 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.723524    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-xks-ca\") pod \"8428ea83-43a2-4f98-a98c-2ac754528306\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") "
Oct 31 02:14:02.723742 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.723725    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-openshift-ca\") pod \"8428ea83-43a2-4f98-a98c-2ac754528306\" (UID: \"8428ea83-43a2-4f98-a98c-2ac754528306\") "
Oct 31 02:14:02.724159 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.724127    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-openshift-ca" pod "8428ea83-43a2-4f98-a98c-2ac754528306" (UID: "8428ea83-43a2-4f98-a98c-2ac754528306"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:02.724518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.724498    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:02.725821 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.725786    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-xks-ca" pod "8428ea83-43a2-4f98-a98c-2ac754528306" (UID: "8428ea83-43a2-4f98-a98c-2ac754528306"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:02.726182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.726158    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8428ea83-43a2-4f98-a98c-2ac754528306-kube-api-access-492s2" pod "8428ea83-43a2-4f98-a98c-2ac754528306" (UID: "8428ea83-43a2-4f98-a98c-2ac754528306"). InnerVolumeSpecName "kube-api-access-492s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:02.826642 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.826548    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492s2\" (UniqueName: \"kubernetes.io/projected/8428ea83-43a2-4f98-a98c-2ac754528306-kube-api-access-492s2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:02.826642 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:02.826580    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8428ea83-43a2-4f98-a98c-2ac754528306-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:03.206176 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.206138    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8" podUID="a2c566c7-8fa5-4e50-80e0-127bd356d02d" containerName="payload-processing" containerID="cri-o://2322b1956836fb4cb689825b0e09944caace83b5524dedb9e356598306488584" gracePeriod=25
Oct 31 02:14:03.384394 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.384361    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"
Oct 31 02:14:03.384394 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.384380    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b" event={"ID":"8428ea83-43a2-4f98-a98c-2ac754528306","Type":"ContainerDied","Data":"e613b9801e279491ce6cffafa5f9731b8845fc39ac0355f7e02aba98e9d5ca63"}
Oct 31 02:14:03.384673 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.384439    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e613b9801e279491ce6cffafa5f9731b8845fc39ac0355f7e02aba98e9d5ca63"
Oct 31 02:14:03.386501 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.386472    5293 generic.go:356] "Generic (PLEG): container finished" podID="a2c566c7-8fa5-4e50-80e0-127bd356d02d" containerID="2322b1956836fb4cb689825b0e09944caace83b5524dedb9e356598306488584" exitCode=0
Oct 31 02:14:03.386640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.386542    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8" event={"ID":"a2c566c7-8fa5-4e50-80e0-127bd356d02d","Type":"ContainerDied","Data":"2322b1956836fb4cb689825b0e09944caace83b5524dedb9e356598306488584"}
Oct 31 02:14:03.479225 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.479201    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:14:03.497692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.497664    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"]
Oct 31 02:14:03.498468 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498451    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main"
Oct 31 02:14:03.498535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498471    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main"
Oct 31 02:14:03.498535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498485    5293 cpu_manager.go:414] "containerMap: removing container" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy"
Oct 31 02:14:03.498535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498489    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy"
Oct 31 02:14:03.498535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498505    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8428ea83-43a2-4f98-a98c-2ac754528306" containerName="revoke-keys"
Oct 31 02:14:03.498535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498511    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8428ea83-43a2-4f98-a98c-2ac754528306" containerName="revoke-keys"
Oct 31 02:14:03.498535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498522    5293 cpu_manager.go:414] "containerMap: removing container" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main"
Oct 31 02:14:03.498535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498527    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498542    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a2c566c7-8fa5-4e50-80e0-127bd356d02d" containerName="payload-processing"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498547    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c566c7-8fa5-4e50-80e0-127bd356d02d" containerName="payload-processing"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498565    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498569    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498676    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6817381f-1ab2-4b8b-b580-d3b37ee07f6d" containerName="main"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498684    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="10df5a77-8db8-41cb-866d-494e7832c0d1" containerName="istio-proxy"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498691    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8428ea83-43a2-4f98-a98c-2ac754528306" containerName="revoke-keys"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498698    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef71ae1-dac7-470a-941d-e3c2f83f6940" containerName="main"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498706    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c566c7-8fa5-4e50-80e0-127bd356d02d" containerName="payload-processing"
Oct 31 02:14:03.498769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.498714    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cc03be-968e-4464-9710-4d082d728072" containerName="main"
Oct 31 02:14:03.502378 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.502357    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.514994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.514969    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:14:03.515180 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.515032    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:14:03.518290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.518265    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"]
Oct 31 02:14:03.534162 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.534124    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"]
Oct 31 02:14:03.534418 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.534245    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.537545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.537109    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"]
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.539993    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-routing-overlay-volume\") pod \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") "
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.540215    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-kube-api-access-gw447\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-kube-api-access-gw447\") pod \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") "
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.540296    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-plugins-config-volume\") pod \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") "
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.540347    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-provider-credentials\") pod \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\" (UID: \"a2c566c7-8fa5-4e50-80e0-127bd356d02d\") "
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.540746    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.540853    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqbd\" (UniqueName: \"kubernetes.io/projected/70bca3a4-8f28-461a-a742-d6a2c47afcef-kube-api-access-qpqbd\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.541023    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzzb\" (UniqueName: \"kubernetes.io/projected/23f13b80-a936-4d63-aa4b-7a97204b84bc-kube-api-access-gfzzb\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.541127    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.541158    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.541227    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.541773    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-routing-overlay-volume" pod "a2c566c7-8fa5-4e50-80e0-127bd356d02d" (UID: "a2c566c7-8fa5-4e50-80e0-127bd356d02d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:03.544070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.542510    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-plugins-config-volume" pod "a2c566c7-8fa5-4e50-80e0-127bd356d02d" (UID: "a2c566c7-8fa5-4e50-80e0-127bd356d02d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:03.548149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.548113    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-kube-api-access-gw447" pod "a2c566c7-8fa5-4e50-80e0-127bd356d02d" (UID: "a2c566c7-8fa5-4e50-80e0-127bd356d02d"). InnerVolumeSpecName "kube-api-access-gw447". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:03.548476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.548447    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-provider-credentials" pod "a2c566c7-8fa5-4e50-80e0-127bd356d02d" (UID: "a2c566c7-8fa5-4e50-80e0-127bd356d02d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:03.642548 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.642504    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqbd\" (UniqueName: \"kubernetes.io/projected/70bca3a4-8f28-461a-a742-d6a2c47afcef-kube-api-access-qpqbd\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.642727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.642588    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzzb\" (UniqueName: \"kubernetes.io/projected/23f13b80-a936-4d63-aa4b-7a97204b84bc-kube-api-access-gfzzb\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.642727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.642660    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.642727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.642683    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.642908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.642747    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.642996 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.642977    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.643153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.643138    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw447\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-kube-api-access-gw447\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:03.643242 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.643158    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:03.643242 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.643173    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c566c7-8fa5-4e50-80e0-127bd356d02d-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:03.643242 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.643191    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c566c7-8fa5-4e50-80e0-127bd356d02d-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:03.643448 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.643425    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.643704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.643686    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.645958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.645933    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.646088 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.645933    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.656644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.656620    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzzb\" (UniqueName: \"kubernetes.io/projected/23f13b80-a936-4d63-aa4b-7a97204b84bc-kube-api-access-gfzzb\") pod \"maas-api-revoke-sub-2a020f30-rsthb\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.656728 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.656618    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqbd\" (UniqueName: \"kubernetes.io/projected/70bca3a4-8f28-461a-a742-d6a2c47afcef-kube-api-access-qpqbd\") pod \"maas-api-revoke-sub-2e2fbf65-95bkj\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:03.855155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.846330    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:03.855155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.846712    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp" podUID="84b6bee3-6b9b-402f-b18a-43418d9ce3ff" containerName="payload-processing" containerID="cri-o://de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3" gracePeriod=25
Oct 31 02:14:03.857386 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:03.857315    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:04.194943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.194899    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:14:04.246860 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.246825    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"]
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.256161    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-routing-overlay-volume\") pod \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") "
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.256208    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-provider-credentials\") pod \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") "
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.256492    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-plugins-config-volume\") pod \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") "
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.256495    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-routing-overlay-volume" pod "84b6bee3-6b9b-402f-b18a-43418d9ce3ff" (UID: "84b6bee3-6b9b-402f-b18a-43418d9ce3ff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.256608    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-kube-api-access-pzbx6\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-kube-api-access-pzbx6\") pod \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\" (UID: \"84b6bee3-6b9b-402f-b18a-43418d9ce3ff\") "
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.256903    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-plugins-config-volume" pod "84b6bee3-6b9b-402f-b18a-43418d9ce3ff" (UID: "84b6bee3-6b9b-402f-b18a-43418d9ce3ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.257528    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"]
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.257577    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:04.258975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.257602    5293 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-routing-overlay-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:04.260110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.260085    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-provider-credentials" pod "84b6bee3-6b9b-402f-b18a-43418d9ce3ff" (UID: "84b6bee3-6b9b-402f-b18a-43418d9ce3ff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:04.260419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.260396    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-kube-api-access-pzbx6" pod "84b6bee3-6b9b-402f-b18a-43418d9ce3ff" (UID: "84b6bee3-6b9b-402f-b18a-43418d9ce3ff"). InnerVolumeSpecName "kube-api-access-pzbx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:04.261146 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:14:04.261122    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f13b80_a936_4d63_aa4b_7a97204b84bc.slice/crio-04b30e56385389e64422f7a010831a4f9b19cc00d168d972112722b3a2940c54 WatchSource:0}: Error finding container 04b30e56385389e64422f7a010831a4f9b19cc00d168d972112722b3a2940c54: Status 404 returned error can't find the container with id 04b30e56385389e64422f7a010831a4f9b19cc00d168d972112722b3a2940c54
Oct 31 02:14:04.359020 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.358991    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzbx6\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-kube-api-access-pzbx6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:04.359020 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.359020    5293 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b6bee3-6b9b-402f-b18a-43418d9ce3ff-provider-credentials\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:04.398126 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.398090    5293 generic.go:356] "Generic (PLEG): container finished" podID="84b6bee3-6b9b-402f-b18a-43418d9ce3ff" containerID="de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3" exitCode=0
Oct 31 02:14:04.398294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.398146    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"
Oct 31 02:14:04.398294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.398192    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp" event={"ID":"84b6bee3-6b9b-402f-b18a-43418d9ce3ff","Type":"ContainerDied","Data":"de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3"}
Oct 31 02:14:04.398294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.398253    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp" event={"ID":"84b6bee3-6b9b-402f-b18a-43418d9ce3ff","Type":"ContainerDied","Data":"293d07e85d93bb5b4c8f8130dea95e9a17d7610a488ff2a44ab2beebb75c015c"}
Oct 31 02:14:04.398294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.398281    5293 scope.go:122] "RemoveContainer" containerID="de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3"
Oct 31 02:14:04.400379 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.400353    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb" event={"ID":"23f13b80-a936-4d63-aa4b-7a97204b84bc","Type":"ContainerStarted","Data":"04b30e56385389e64422f7a010831a4f9b19cc00d168d972112722b3a2940c54"}
Oct 31 02:14:04.407223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.407204    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj" event={"ID":"70bca3a4-8f28-461a-a742-d6a2c47afcef","Type":"ContainerStarted","Data":"0568f694f791289772500910dde4afab38a7e702d87abd6df97cc673bde31fa2"}
Oct 31 02:14:04.409454 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.409404    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8" event={"ID":"a2c566c7-8fa5-4e50-80e0-127bd356d02d","Type":"ContainerDied","Data":"27231d23028d10ce51529e124a3bf3a6471d31a1a3bd25e34cee28867e59ff19"}
Oct 31 02:14:04.409548 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.409456    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"
Oct 31 02:14:04.412689 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.412672    5293 scope.go:122] "RemoveContainer" containerID="de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3"
Oct 31 02:14:04.412939 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:14:04.412918    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3\": container with ID starting with de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3 not found: ID does not exist" containerID="de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3"
Oct 31 02:14:04.413016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.412952    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3"} err="failed to get container status \"de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3\": rpc error: code = NotFound desc = could not find container \"de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3\": container with ID starting with de1f78c095ee5cca1bb35b5e8841024a1bd108bb942e11673b7d271b87b5cff3 not found: ID does not exist"
Oct 31 02:14:04.413016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.412975    5293 scope.go:122] "RemoveContainer" containerID="2322b1956836fb4cb689825b0e09944caace83b5524dedb9e356598306488584"
Oct 31 02:14:04.439981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.439914    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"]
Oct 31 02:14:04.443809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.443782    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6zxqtp"]
Oct 31 02:14:04.448679 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.448655    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"]
Oct 31 02:14:04.452497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:04.452471    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688b5zw8"]
Oct 31 02:14:05.440093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:05.440018    5293 generic.go:356] "Generic (PLEG): container finished" podID="23f13b80-a936-4d63-aa4b-7a97204b84bc" containerID="f455bd29bc306f449b61baf191f04227dba1774ac700ba69e5ad13be2c071234" exitCode=0
Oct 31 02:14:05.440584 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:05.440095    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb" event={"ID":"23f13b80-a936-4d63-aa4b-7a97204b84bc","Type":"ContainerDied","Data":"f455bd29bc306f449b61baf191f04227dba1774ac700ba69e5ad13be2c071234"}
Oct 31 02:14:05.442772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:05.442746    5293 generic.go:356] "Generic (PLEG): container finished" podID="70bca3a4-8f28-461a-a742-d6a2c47afcef" containerID="596ddf8824c6dcc2ac76dabef4bac973fad1c09532f90ef12c30d262b834f15f" exitCode=0
Oct 31 02:14:05.442878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:05.442869    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj" event={"ID":"70bca3a4-8f28-461a-a742-d6a2c47afcef","Type":"ContainerDied","Data":"596ddf8824c6dcc2ac76dabef4bac973fad1c09532f90ef12c30d262b834f15f"}
Oct 31 02:14:06.201937 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.201904    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b6bee3-6b9b-402f-b18a-43418d9ce3ff" path="/var/lib/kubelet/pods/84b6bee3-6b9b-402f-b18a-43418d9ce3ff/volumes"
Oct 31 02:14:06.202468 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.202452    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c566c7-8fa5-4e50-80e0-127bd356d02d" path="/var/lib/kubelet/pods/a2c566c7-8fa5-4e50-80e0-127bd356d02d/volumes"
Oct 31 02:14:06.686458 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.686430    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:06.688521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.688494    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:06.731093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.731014    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f13b80-a936-4d63-aa4b-7a97204b84bc-kube-api-access-gfzzb\" (UniqueName: \"kubernetes.io/projected/23f13b80-a936-4d63-aa4b-7a97204b84bc-kube-api-access-gfzzb\") pod \"23f13b80-a936-4d63-aa4b-7a97204b84bc\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") "
Oct 31 02:14:06.731093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.731084    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-xks-ca\") pod \"23f13b80-a936-4d63-aa4b-7a97204b84bc\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") "
Oct 31 02:14:06.731364 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.731114    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-openshift-ca\") pod \"70bca3a4-8f28-461a-a742-d6a2c47afcef\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") "
Oct 31 02:14:06.731364 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.731262    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bca3a4-8f28-461a-a742-d6a2c47afcef-kube-api-access-qpqbd\" (UniqueName: \"kubernetes.io/projected/70bca3a4-8f28-461a-a742-d6a2c47afcef-kube-api-access-qpqbd\") pod \"70bca3a4-8f28-461a-a742-d6a2c47afcef\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") "
Oct 31 02:14:06.731833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.731405    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-openshift-ca\") pod \"23f13b80-a936-4d63-aa4b-7a97204b84bc\" (UID: \"23f13b80-a936-4d63-aa4b-7a97204b84bc\") "
Oct 31 02:14:06.731985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.731832    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-openshift-ca" pod "23f13b80-a936-4d63-aa4b-7a97204b84bc" (UID: "23f13b80-a936-4d63-aa4b-7a97204b84bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:06.731985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.731929    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-xks-ca\") pod \"70bca3a4-8f28-461a-a742-d6a2c47afcef\" (UID: \"70bca3a4-8f28-461a-a742-d6a2c47afcef\") "
Oct 31 02:14:06.732737 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.732715    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:06.734166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.734139    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-openshift-ca" pod "70bca3a4-8f28-461a-a742-d6a2c47afcef" (UID: "70bca3a4-8f28-461a-a742-d6a2c47afcef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:06.736962 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.736931    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-xks-ca" pod "70bca3a4-8f28-461a-a742-d6a2c47afcef" (UID: "70bca3a4-8f28-461a-a742-d6a2c47afcef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:06.737294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.737260    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-xks-ca" pod "23f13b80-a936-4d63-aa4b-7a97204b84bc" (UID: "23f13b80-a936-4d63-aa4b-7a97204b84bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:06.737952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.737929    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bca3a4-8f28-461a-a742-d6a2c47afcef-kube-api-access-qpqbd" pod "70bca3a4-8f28-461a-a742-d6a2c47afcef" (UID: "70bca3a4-8f28-461a-a742-d6a2c47afcef"). InnerVolumeSpecName "kube-api-access-qpqbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:06.739612 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.739577    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f13b80-a936-4d63-aa4b-7a97204b84bc-kube-api-access-gfzzb" pod "23f13b80-a936-4d63-aa4b-7a97204b84bc" (UID: "23f13b80-a936-4d63-aa4b-7a97204b84bc"). InnerVolumeSpecName "kube-api-access-gfzzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:06.834304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.834272    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqbd\" (UniqueName: \"kubernetes.io/projected/70bca3a4-8f28-461a-a742-d6a2c47afcef-kube-api-access-qpqbd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:06.834304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.834300    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:06.834304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.834309    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfzzb\" (UniqueName: \"kubernetes.io/projected/23f13b80-a936-4d63-aa4b-7a97204b84bc-kube-api-access-gfzzb\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:06.834531 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.834318    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f13b80-a936-4d63-aa4b-7a97204b84bc-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:06.834531 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:06.834327    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bca3a4-8f28-461a-a742-d6a2c47afcef-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:07.462682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:07.462654    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"
Oct 31 02:14:07.462908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:07.462653    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb" event={"ID":"23f13b80-a936-4d63-aa4b-7a97204b84bc","Type":"ContainerDied","Data":"04b30e56385389e64422f7a010831a4f9b19cc00d168d972112722b3a2940c54"}
Oct 31 02:14:07.462908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:07.462781    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04b30e56385389e64422f7a010831a4f9b19cc00d168d972112722b3a2940c54"
Oct 31 02:14:07.465593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:07.465571    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"
Oct 31 02:14:07.465733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:07.465619    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj" event={"ID":"70bca3a4-8f28-461a-a742-d6a2c47afcef","Type":"ContainerDied","Data":"0568f694f791289772500910dde4afab38a7e702d87abd6df97cc673bde31fa2"}
Oct 31 02:14:07.465733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:07.465664    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0568f694f791289772500910dde4afab38a7e702d87abd6df97cc673bde31fa2"
Oct 31 02:14:11.779800 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:14:11.779763    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:14:19.962778 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:19.962738    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"]
Oct 31 02:14:19.980459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:19.980405    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be6935-nchvx"]
Oct 31 02:14:20.218743 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:20.218643    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f264f44-9b2a-405f-bb7a-62e769613363" path="/var/lib/kubelet/pods/0f264f44-9b2a-405f-bb7a-62e769613363/volumes"
Oct 31 02:14:25.317201 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:25.317165    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:14:28.652134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:28.652098    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"]
Oct 31 02:14:28.671061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:28.653000    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" containerID="cri-o://8232cc6607897a904f877232a4792cb6d90322625f4fa4537fdb27b9f920499f" gracePeriod=30
Oct 31 02:14:28.811065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:28.806559    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"]
Oct 31 02:14:28.811065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:28.806820    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" containerID="cri-o://3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5" gracePeriod=30
Oct 31 02:14:30.327566 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:30.327302    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:14:30.327566 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:30.327380    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:14:30.670061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:30.664149    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:14:30.797064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:30.796222    5293 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:14:30.797064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:30.796302    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:14:31.247061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:31.236239    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Oct 31 02:14:31.287171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:31.287121    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:14:32.139251 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:32.139209    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"]
Oct 31 02:14:32.162577 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:32.162544    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9308413-kk779"]
Oct 31 02:14:32.262065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:32.261154    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819ff781-73a2-4758-af55-e8b8fbcdba3d" path="/var/lib/kubelet/pods/819ff781-73a2-4758-af55-e8b8fbcdba3d/volumes"
Oct 31 02:14:34.155871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.155819    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 02:14:34.157648 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.157622    5293 generic.go:356] "Generic (PLEG): container finished" podID="67644859-8273-4219-b427-970e6e36d505" containerID="3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5" exitCode=0
Oct 31 02:14:34.157966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.157936    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" event={"ID":"67644859-8273-4219-b427-970e6e36d505","Type":"ContainerDied","Data":"3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5"}
Oct 31 02:14:34.158120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.158074    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7" event={"ID":"67644859-8273-4219-b427-970e6e36d505","Type":"ContainerDied","Data":"25639d26779eb61ca583b6aff353203bc94e39f8c226a1b44738bca32c1162a3"}
Oct 31 02:14:34.158120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.158117    5293 scope.go:122] "RemoveContainer" containerID="3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5"
Oct 31 02:14:34.160644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.160620    5293 generic.go:356] "Generic (PLEG): container finished" podID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerID="8232cc6607897a904f877232a4792cb6d90322625f4fa4537fdb27b9f920499f" exitCode=0
Oct 31 02:14:34.160843 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.160788    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" event={"ID":"55aeddbd-9e95-437d-b476-c5fc291339b0","Type":"ContainerDied","Data":"8232cc6607897a904f877232a4792cb6d90322625f4fa4537fdb27b9f920499f"}
Oct 31 02:14:34.187601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.179537    5293 scope.go:122] "RemoveContainer" containerID="3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5"
Oct 31 02:14:34.187601 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:14:34.179845    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5\": container with ID starting with 3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5 not found: ID does not exist" containerID="3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5"
Oct 31 02:14:34.187601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.179881    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5"} err="failed to get container status \"3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5\": rpc error: code = NotFound desc = could not find container \"3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5\": container with ID starting with 3cb073bcfd210412b6c2b189ac6ab4bef34b6333c6aa1b62bf7db2c26e61caa5 not found: ID does not exist"
Oct 31 02:14:34.230896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.230587    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 02:14:34.232592 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.232539    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67644859-8273-4219-b427-970e6e36d505-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644859-8273-4219-b427-970e6e36d505-istiod-ca-cert\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.232761 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.232597    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-istio-token\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-istio-token\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.232761 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.232652    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-certs\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.232887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.232816    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-socket\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.232887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.232848    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67644859-8273-4219-b427-970e6e36d505-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644859-8273-4219-b427-970e6e36d505-istio-podinfo\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.232994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.232902    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-envoy\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.232994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.232943    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-credential-socket\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.233135 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.232991    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-kube-api-access-gz4l6\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-kube-api-access-gz4l6\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.233192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.233165    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-data\") pod \"67644859-8273-4219-b427-970e6e36d505\" (UID: \"67644859-8273-4219-b427-970e6e36d505\") "
Oct 31 02:14:34.233490 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.233468    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-socket" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.233873 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.233854    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.234326 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.234306    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67644859-8273-4219-b427-970e6e36d505-istiod-ca-cert" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:34.235128 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.234835    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-data" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.235339 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.235272    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-certs" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.235486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.235450    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-credential-socket" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.237431 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.237399    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-envoy" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.237959 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.237929    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67644859-8273-4219-b427-970e6e36d505-istio-podinfo" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:14:34.239170 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.239139    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-istio-token" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:34.239508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.239482    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-kube-api-access-gz4l6" pod "67644859-8273-4219-b427-970e6e36d505" (UID: "67644859-8273-4219-b427-970e6e36d505"). InnerVolumeSpecName "kube-api-access-gz4l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:34.335127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335083    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-podinfo\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335185    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-token\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-token\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335253    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-envoy\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335447 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335338    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-data\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335447 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335403    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-kube-api-access-47glx\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-kube-api-access-47glx\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335443    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-certs\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335519    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55aeddbd-9e95-437d-b476-c5fc291339b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aeddbd-9e95-437d-b476-c5fc291339b0-istiod-ca-cert\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335683 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335559    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-credential-socket\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335683 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335626    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-socket\") pod \"55aeddbd-9e95-437d-b476-c5fc291339b0\" (UID: \"55aeddbd-9e95-437d-b476-c5fc291339b0\") "
Oct 31 02:14:34.335940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.335882    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-credential-socket" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.336155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.336089    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-data" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.336380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.336186    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-socket" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.336380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.336299    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55aeddbd-9e95-437d-b476-c5fc291339b0-istiod-ca-cert" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:34.336380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.336366    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-certs" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.338406 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338386    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644859-8273-4219-b427-970e6e36d505-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338412    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338428    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338441    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4l6\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-kube-api-access-gz4l6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338456    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338468    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338481    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aeddbd-9e95-437d-b476-c5fc291339b0-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338494    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338508    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338520    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338534    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644859-8273-4219-b427-970e6e36d505-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338547    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67644859-8273-4219-b427-970e6e36d505-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338561    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644859-8273-4219-b427-970e6e36d505-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.338999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338941    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-podinfo" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:14:34.338999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.338984    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-envoy" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.339248 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.339001    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-kube-api-access-47glx" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "kube-api-access-47glx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:34.339248 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.339030    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-token" pod "55aeddbd-9e95-437d-b476-c5fc291339b0" (UID: "55aeddbd-9e95-437d-b476-c5fc291339b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:34.453082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.449300    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47glx\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-kube-api-access-47glx\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.453082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.449339    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.453082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.449356    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.453082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:34.449371    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aeddbd-9e95-437d-b476-c5fc291339b0-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:35.230510 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:35.230484    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"
Oct 31 02:14:35.242542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:35.242020    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s" event={"ID":"55aeddbd-9e95-437d-b476-c5fc291339b0","Type":"ContainerDied","Data":"351667a55fbf4dd89c8f004a4e1303b9e94e91f51d347e4b5a9cf75c932a47b8"}
Oct 31 02:14:35.242542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:35.242129    5293 scope.go:122] "RemoveContainer" containerID="8232cc6607897a904f877232a4792cb6d90322625f4fa4537fdb27b9f920499f"
Oct 31 02:14:35.242542 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:35.242246    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"
Oct 31 02:14:35.265176 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:35.265150    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"]
Oct 31 02:14:35.269933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:35.269907    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-hlms7"]
Oct 31 02:14:35.280889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:35.280869    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"]
Oct 31 02:14:35.286627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:35.286606    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j782s"]
Oct 31 02:14:36.175665 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:36.175633    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" path="/var/lib/kubelet/pods/55aeddbd-9e95-437d-b476-c5fc291339b0/volumes"
Oct 31 02:14:36.176129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:36.176113    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67644859-8273-4219-b427-970e6e36d505" path="/var/lib/kubelet/pods/67644859-8273-4219-b427-970e6e36d505/volumes"
Oct 31 02:14:39.707427 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:39.707184    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"]
Oct 31 02:14:39.707966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:39.707478    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy" containerID="cri-o://93ea254fdbb5a5d032e80d154e46a029f66eb6ee7d8ecc3503908046fa4c7c42" gracePeriod=30
Oct 31 02:14:40.271100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.270413    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"]
Oct 31 02:14:40.271487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271441    5293 cpu_manager.go:414] "containerMap: removing container" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy"
Oct 31 02:14:40.271487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271464    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy"
Oct 31 02:14:40.271487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271492    5293 cpu_manager.go:414] "containerMap: removing container" podUID="23f13b80-a936-4d63-aa4b-7a97204b84bc" containerName="revoke-keys"
Oct 31 02:14:40.271733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271501    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f13b80-a936-4d63-aa4b-7a97204b84bc" containerName="revoke-keys"
Oct 31 02:14:40.271733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271554    5293 cpu_manager.go:414] "containerMap: removing container" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy"
Oct 31 02:14:40.271733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271563    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy"
Oct 31 02:14:40.271733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271579    5293 cpu_manager.go:414] "containerMap: removing container" podUID="84b6bee3-6b9b-402f-b18a-43418d9ce3ff" containerName="payload-processing"
Oct 31 02:14:40.271733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271586    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b6bee3-6b9b-402f-b18a-43418d9ce3ff" containerName="payload-processing"
Oct 31 02:14:40.271733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271622    5293 cpu_manager.go:414] "containerMap: removing container" podUID="70bca3a4-8f28-461a-a742-d6a2c47afcef" containerName="revoke-keys"
Oct 31 02:14:40.271733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271629    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bca3a4-8f28-461a-a742-d6a2c47afcef" containerName="revoke-keys"
Oct 31 02:14:40.272229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271839    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="55aeddbd-9e95-437d-b476-c5fc291339b0" containerName="istio-proxy"
Oct 31 02:14:40.272229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271858    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f13b80-a936-4d63-aa4b-7a97204b84bc" containerName="revoke-keys"
Oct 31 02:14:40.272229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271867    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="67644859-8273-4219-b427-970e6e36d505" containerName="istio-proxy"
Oct 31 02:14:40.272229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271879    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b6bee3-6b9b-402f-b18a-43418d9ce3ff" containerName="payload-processing"
Oct 31 02:14:40.272229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.271892    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bca3a4-8f28-461a-a742-d6a2c47afcef" containerName="revoke-keys"
Oct 31 02:14:40.286888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.286857    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"]
Oct 31 02:14:40.287113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.286940    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.340625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.340463    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:14:40.340625 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.340496    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:14:40.422135 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.422086    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.422362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.422279    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spwd5\" (UniqueName: \"kubernetes.io/projected/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-kube-api-access-spwd5\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.422417 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.422363    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.523823 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.523735    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwd5\" (UniqueName: \"kubernetes.io/projected/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-kube-api-access-spwd5\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.523823 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.523787    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.524091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.524061    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.524809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.524785    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.526503 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.526478    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.536199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.536163    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spwd5\" (UniqueName: \"kubernetes.io/projected/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-kube-api-access-spwd5\") pod \"maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.613451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.613417    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:40.901439 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:40.901412    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"]
Oct 31 02:14:40.902704 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:14:40.902678    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4ebc0c_bdc0_4272_aa20_6e031fb00027.slice/crio-9d28758ae9691cd0f0871d5c8525c229c7ba4fd47bd8014e0967483d87cc31b4 WatchSource:0}: Error finding container 9d28758ae9691cd0f0871d5c8525c229c7ba4fd47bd8014e0967483d87cc31b4: Status 404 returned error can't find the container with id 9d28758ae9691cd0f0871d5c8525c229c7ba4fd47bd8014e0967483d87cc31b4
Oct 31 02:14:41.299084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:41.299008    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx" event={"ID":"bf4ebc0c-bdc0-4272-aa20-6e031fb00027","Type":"ContainerStarted","Data":"9d28758ae9691cd0f0871d5c8525c229c7ba4fd47bd8014e0967483d87cc31b4"}
Oct 31 02:14:41.781924 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:14:41.781882    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:14:42.308018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:42.307984    5293 generic.go:356] "Generic (PLEG): container finished" podID="bf4ebc0c-bdc0-4272-aa20-6e031fb00027" containerID="1d47b1ea5ea1000fb0688c336f6e30ad9e41be108dec88bcc2e51a0f741b9c0a" exitCode=0
Oct 31 02:14:42.308437 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:42.308067    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx" event={"ID":"bf4ebc0c-bdc0-4272-aa20-6e031fb00027","Type":"ContainerDied","Data":"1d47b1ea5ea1000fb0688c336f6e30ad9e41be108dec88bcc2e51a0f741b9c0a"}
Oct 31 02:14:43.025145 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.025114    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"]
Oct 31 02:14:43.035440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.035410    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47681e17-nhxf7"]
Oct 31 02:14:43.492958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.492934    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:43.540002 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.539971    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-kube-api-access-spwd5\" (UniqueName: \"kubernetes.io/projected/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-kube-api-access-spwd5\") pod \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") "
Oct 31 02:14:43.540189 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.540065    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-openshift-ca\") pod \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") "
Oct 31 02:14:43.540238 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.540186    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-xks-ca\") pod \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\" (UID: \"bf4ebc0c-bdc0-4272-aa20-6e031fb00027\") "
Oct 31 02:14:43.540580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.540549    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-openshift-ca" pod "bf4ebc0c-bdc0-4272-aa20-6e031fb00027" (UID: "bf4ebc0c-bdc0-4272-aa20-6e031fb00027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:43.540961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.540937    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:43.542527 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.542501    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-kube-api-access-spwd5" pod "bf4ebc0c-bdc0-4272-aa20-6e031fb00027" (UID: "bf4ebc0c-bdc0-4272-aa20-6e031fb00027"). InnerVolumeSpecName "kube-api-access-spwd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:43.542988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.542963    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-xks-ca" pod "bf4ebc0c-bdc0-4272-aa20-6e031fb00027" (UID: "bf4ebc0c-bdc0-4272-aa20-6e031fb00027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:43.642149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.642075    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spwd5\" (UniqueName: \"kubernetes.io/projected/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-kube-api-access-spwd5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:43.642149 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:43.642107    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4ebc0c-bdc0-4272-aa20-6e031fb00027-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:44.176169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:44.176129    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2722d7a2-227c-4d59-b497-1d48c4aa896b" path="/var/lib/kubelet/pods/2722d7a2-227c-4d59-b497-1d48c4aa896b/volumes"
Oct 31 02:14:44.328586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:44.328560    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"
Oct 31 02:14:44.328795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:44.328601    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx" event={"ID":"bf4ebc0c-bdc0-4272-aa20-6e031fb00027","Type":"ContainerDied","Data":"9d28758ae9691cd0f0871d5c8525c229c7ba4fd47bd8014e0967483d87cc31b4"}
Oct 31 02:14:44.328795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:44.328661    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d28758ae9691cd0f0871d5c8525c229c7ba4fd47bd8014e0967483d87cc31b4"
Oct 31 02:14:45.346024 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.345993    5293 generic.go:356] "Generic (PLEG): container finished" podID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerID="93ea254fdbb5a5d032e80d154e46a029f66eb6ee7d8ecc3503908046fa4c7c42" exitCode=0
Oct 31 02:14:45.346580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.346178    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" event={"ID":"cc999384-bcd7-4dda-95ca-4ae3bf4fd130","Type":"ContainerDied","Data":"93ea254fdbb5a5d032e80d154e46a029f66eb6ee7d8ecc3503908046fa4c7c42"}
Oct 31 02:14:45.346580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.346233    5293 scope.go:122] "RemoveContainer" containerID="1e8ad41d70f7333d215af8be4fff2feac8bc2168978c52af51c46036066df662"
Oct 31 02:14:45.471802 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.471777    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 02:14:45.584421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584329    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istiod-ca-cert\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.584421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584375    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-kube-api-access-wfvsc\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-kube-api-access-wfvsc\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.584659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584456    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-podinfo\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.584659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584502    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-data\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.584659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584558    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-socket\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.584659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584603    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-token\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-token\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.584878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584738    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-envoy\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.585243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584765    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istiod-ca-cert" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:45.585243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584888    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-socket" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:45.585621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584902    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-data" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:45.585621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.585108    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-certs" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:45.585621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.584830    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-certs\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.585621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.585483    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-credential-socket\") pod \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\" (UID: \"cc999384-bcd7-4dda-95ca-4ae3bf4fd130\") "
Oct 31 02:14:45.586546 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.586421    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:45.586546 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.586450    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:45.586546 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.586468    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:45.586546 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.586484    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:45.586843 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.586548    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-credential-socket" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:45.587418 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.587393    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-kube-api-access-wfvsc" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "kube-api-access-wfvsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:45.587534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.587481    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-envoy" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:45.587702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.587686    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-podinfo" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:14:45.587857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.587835    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-token" pod "cc999384-bcd7-4dda-95ca-4ae3bf4fd130" (UID: "cc999384-bcd7-4dda-95ca-4ae3bf4fd130"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:45.687857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.687825    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:45.687857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.687852    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:45.687857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.687862    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvsc\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-kube-api-access-wfvsc\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:45.688194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.687872    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:45.688194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:45.687880    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc999384-bcd7-4dda-95ca-4ae3bf4fd130-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:46.358000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:46.357962    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"
Oct 31 02:14:46.358459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:46.357962    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl" event={"ID":"cc999384-bcd7-4dda-95ca-4ae3bf4fd130","Type":"ContainerDied","Data":"dc34448bee719704c728a3c8f095241eaeb9820909aeb57038a210c3078bdf79"}
Oct 31 02:14:46.358459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:46.358124    5293 scope.go:122] "RemoveContainer" containerID="93ea254fdbb5a5d032e80d154e46a029f66eb6ee7d8ecc3503908046fa4c7c42"
Oct 31 02:14:46.381103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:46.381072    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"]
Oct 31 02:14:46.384692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:46.384669    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ac7e0d-openshift-default-6b54b7bd69-lrrcl"]
Oct 31 02:14:48.174706 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:48.174677    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" path="/var/lib/kubelet/pods/cc999384-bcd7-4dda-95ca-4ae3bf4fd130/volumes"
Oct 31 02:14:50.383207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:50.383173    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"]
Oct 31 02:14:50.436640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:50.436607    5293 scope.go:122] "RemoveContainer" containerID="60c7b614bc63b229a7292fee9f7ec979ae8c26be6769a5d703a594dd7b5ec5d2"
Oct 31 02:14:51.552067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:51.547234    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"]
Oct 31 02:14:51.563733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:51.559696    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0b6abf-29bgx"]
Oct 31 02:14:52.175096 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:52.175006    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4ebc0c-bdc0-4272-aa20-6e031fb00027" path="/var/lib/kubelet/pods/bf4ebc0c-bdc0-4272-aa20-6e031fb00027/volumes"
Oct 31 02:14:55.383874 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:14:55.383837    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn" podUID="43e2b864-2259-4446-89ad-0052cc9519b7" containerName="payload-processing" containerID="cri-o://92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc" gracePeriod=25
Oct 31 02:15:00.170459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.170412    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"]
Oct 31 02:15:00.171594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171562    5293 cpu_manager.go:414] "containerMap: removing container" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy"
Oct 31 02:15:00.171594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171591    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy"
Oct 31 02:15:00.171828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171606    5293 cpu_manager.go:414] "containerMap: removing container" podUID="bf4ebc0c-bdc0-4272-aa20-6e031fb00027" containerName="revoke-keys"
Oct 31 02:15:00.171828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171614    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4ebc0c-bdc0-4272-aa20-6e031fb00027" containerName="revoke-keys"
Oct 31 02:15:00.171828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171653    5293 cpu_manager.go:414] "containerMap: removing container" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy"
Oct 31 02:15:00.171828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171661    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy"
Oct 31 02:15:00.172082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171837    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy"
Oct 31 02:15:00.172082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171863    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc999384-bcd7-4dda-95ca-4ae3bf4fd130" containerName="istio-proxy"
Oct 31 02:15:00.172082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.171874    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4ebc0c-bdc0-4272-aa20-6e031fb00027" containerName="revoke-keys"
Oct 31 02:15:00.205965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.205936    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:00.260644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.260611    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"]
Oct 31 02:15:00.287126 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.275464    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9j9\" (UniqueName: \"kubernetes.io/projected/9b53e41f-48b3-4e5a-b600-8bca52c3526c-kube-api-access-zx9j9\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh\" (UID: \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:00.320080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.283073    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:15:00.320080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.287677    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"]
Oct 31 02:15:00.320080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.291139    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:00.320080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.293171    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"]
Oct 31 02:15:00.320080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.297376    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:00.340196 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.340030    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b53e41f-48b3-4e5a-b600-8bca52c3526c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh\" (UID: \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:00.352305 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.352276    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"]
Oct 31 02:15:00.356317 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.356024    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:00.386194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.383973    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"]
Oct 31 02:15:00.386194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.384029    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"]
Oct 31 02:15:00.386194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.384077    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"]
Oct 31 02:15:00.386878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.386731    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:00.411118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.392373    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"]
Oct 31 02:15:00.411118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.392526    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"]
Oct 31 02:15:00.411118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.392468    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:00.411118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.392632    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"]
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.470107    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b53e41f-48b3-4e5a-b600-8bca52c3526c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh\" (UID: \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.470190    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghj8v\" (UniqueName: \"kubernetes.io/projected/24a60988-774c-48a9-9a51-437c0e70c633-kube-api-access-ghj8v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj\" (UID: \"24a60988-774c-48a9-9a51-437c0e70c633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.470454    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a3759-7be6-416a-97bc-d2d5e6e96685-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr\" (UID: \"575a3759-7be6-416a-97bc-d2d5e6e96685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.470488    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9j9\" (UniqueName: \"kubernetes.io/projected/9b53e41f-48b3-4e5a-b600-8bca52c3526c-kube-api-access-zx9j9\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh\" (UID: \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.470535    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjw4r\" (UniqueName: \"kubernetes.io/projected/117b4c77-4f1d-4b01-8665-f641b77b757c-kube-api-access-zjw4r\") pod \"maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc\" (UID: \"117b4c77-4f1d-4b01-8665-f641b77b757c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.470628    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a60988-774c-48a9-9a51-437c0e70c633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj\" (UID: \"24a60988-774c-48a9-9a51-437c0e70c633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.470666    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117b4c77-4f1d-4b01-8665-f641b77b757c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc\" (UID: \"117b4c77-4f1d-4b01-8665-f641b77b757c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.470692    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4k6j\" (UniqueName: \"kubernetes.io/projected/575a3759-7be6-416a-97bc-d2d5e6e96685-kube-api-access-p4k6j\") pod \"maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr\" (UID: \"575a3759-7be6-416a-97bc-d2d5e6e96685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:00.518111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.472302    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b53e41f-48b3-4e5a-b600-8bca52c3526c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh\" (UID: \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.574699    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"]
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.574751    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"]
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.581845    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584082    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"]
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584650    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37eeab3-7312-4381-b0cb-55968b82c051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw\" (UID: \"a37eeab3-7312-4381-b0cb-55968b82c051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584706    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc483cfa-c134-4176-b6b3-71af38093a2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz\" (UID: \"dc483cfa-c134-4176-b6b3-71af38093a2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584744    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt8wh\" (UniqueName: \"kubernetes.io/projected/a37eeab3-7312-4381-b0cb-55968b82c051-kube-api-access-dt8wh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw\" (UID: \"a37eeab3-7312-4381-b0cb-55968b82c051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584820    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a3759-7be6-416a-97bc-d2d5e6e96685-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr\" (UID: \"575a3759-7be6-416a-97bc-d2d5e6e96685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584865    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjw4r\" (UniqueName: \"kubernetes.io/projected/117b4c77-4f1d-4b01-8665-f641b77b757c-kube-api-access-zjw4r\") pod \"maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc\" (UID: \"117b4c77-4f1d-4b01-8665-f641b77b757c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584904    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j\" (UID: \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584953    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5ls2\" (UniqueName: \"kubernetes.io/projected/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-kube-api-access-h5ls2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j\" (UID: \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.584988    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a60988-774c-48a9-9a51-437c0e70c633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj\" (UID: \"24a60988-774c-48a9-9a51-437c0e70c633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.585021    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117b4c77-4f1d-4b01-8665-f641b77b757c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc\" (UID: \"117b4c77-4f1d-4b01-8665-f641b77b757c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.585066    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4k6j\" (UniqueName: \"kubernetes.io/projected/575a3759-7be6-416a-97bc-d2d5e6e96685-kube-api-access-p4k6j\") pod \"maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr\" (UID: \"575a3759-7be6-416a-97bc-d2d5e6e96685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.585135    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr67t\" (UniqueName: \"kubernetes.io/projected/dc483cfa-c134-4176-b6b3-71af38093a2f-kube-api-access-kr67t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz\" (UID: \"dc483cfa-c134-4176-b6b3-71af38093a2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.585203    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghj8v\" (UniqueName: \"kubernetes.io/projected/24a60988-774c-48a9-9a51-437c0e70c633-kube-api-access-ghj8v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj\" (UID: \"24a60988-774c-48a9-9a51-437c0e70c633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.586736    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a60988-774c-48a9-9a51-437c0e70c633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj\" (UID: \"24a60988-774c-48a9-9a51-437c0e70c633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.587835    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117b4c77-4f1d-4b01-8665-f641b77b757c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc\" (UID: \"117b4c77-4f1d-4b01-8665-f641b77b757c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.589283    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a3759-7be6-416a-97bc-d2d5e6e96685-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr\" (UID: \"575a3759-7be6-416a-97bc-d2d5e6e96685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:00.599059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.590454    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:00.600291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.600207    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"]
Oct 31 02:15:00.600291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.600249    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"]
Oct 31 02:15:00.600291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.600277    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"]
Oct 31 02:15:00.600449 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.600303    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"]
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.603126    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.628294    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9j9\" (UniqueName: \"kubernetes.io/projected/9b53e41f-48b3-4e5a-b600-8bca52c3526c-kube-api-access-zx9j9\") pod \"maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh\" (UID: \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.654327    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"]
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.654375    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"]
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.654416    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"]
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.655244    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" containerID="cri-o://a50fec06a82ce909922661733c8c00dac1ced094df28ea4a51691c862045072e" gracePeriod=30
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.660225    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.681481    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghj8v\" (UniqueName: \"kubernetes.io/projected/24a60988-774c-48a9-9a51-437c0e70c633-kube-api-access-ghj8v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj\" (UID: \"24a60988-774c-48a9-9a51-437c0e70c633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.697723    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5ls2\" (UniqueName: \"kubernetes.io/projected/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-kube-api-access-h5ls2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j\" (UID: \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.697790    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40530b75-ac11-4d5c-ad11-3e4f5726d972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5\" (UID: \"40530b75-ac11-4d5c-ad11-3e4f5726d972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.697861    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d96f53-8b5e-4f32-adba-ac197a54931f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc\" (UID: \"29d96f53-8b5e-4f32-adba-ac197a54931f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.699902    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr67t\" (UniqueName: \"kubernetes.io/projected/dc483cfa-c134-4176-b6b3-71af38093a2f-kube-api-access-kr67t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz\" (UID: \"dc483cfa-c134-4176-b6b3-71af38093a2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.699965    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgrd6\" (UniqueName: \"kubernetes.io/projected/40530b75-ac11-4d5c-ad11-3e4f5726d972-kube-api-access-hgrd6\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5\" (UID: \"40530b75-ac11-4d5c-ad11-3e4f5726d972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.700067    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bq9\" (UniqueName: \"kubernetes.io/projected/29d96f53-8b5e-4f32-adba-ac197a54931f-kube-api-access-v8bq9\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc\" (UID: \"29d96f53-8b5e-4f32-adba-ac197a54931f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.700181    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37eeab3-7312-4381-b0cb-55968b82c051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw\" (UID: \"a37eeab3-7312-4381-b0cb-55968b82c051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.700215    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e42c7-d074-49ed-9355-b9cd77d0aace-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j\" (UID: \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.700251    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc483cfa-c134-4176-b6b3-71af38093a2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz\" (UID: \"dc483cfa-c134-4176-b6b3-71af38093a2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.700292    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt8wh\" (UniqueName: \"kubernetes.io/projected/a37eeab3-7312-4381-b0cb-55968b82c051-kube-api-access-dt8wh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw\" (UID: \"a37eeab3-7312-4381-b0cb-55968b82c051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.700627    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hh9\" (UniqueName: \"kubernetes.io/projected/7e8e42c7-d074-49ed-9355-b9cd77d0aace-kube-api-access-24hh9\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j\" (UID: \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:00.701063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.700725    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j\" (UID: \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:00.706989 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.701688    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j\" (UID: \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:00.706989 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.703495    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjw4r\" (UniqueName: \"kubernetes.io/projected/117b4c77-4f1d-4b01-8665-f641b77b757c-kube-api-access-zjw4r\") pod \"maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc\" (UID: \"117b4c77-4f1d-4b01-8665-f641b77b757c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:00.706989 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.705016    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc483cfa-c134-4176-b6b3-71af38093a2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz\" (UID: \"dc483cfa-c134-4176-b6b3-71af38093a2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:00.706989 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.705175    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37eeab3-7312-4381-b0cb-55968b82c051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw\" (UID: \"a37eeab3-7312-4381-b0cb-55968b82c051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:00.708338 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.708317    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4k6j\" (UniqueName: \"kubernetes.io/projected/575a3759-7be6-416a-97bc-d2d5e6e96685-kube-api-access-p4k6j\") pod \"maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr\" (UID: \"575a3759-7be6-416a-97bc-d2d5e6e96685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:00.714970 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.714939    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt8wh\" (UniqueName: \"kubernetes.io/projected/a37eeab3-7312-4381-b0cb-55968b82c051-kube-api-access-dt8wh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw\" (UID: \"a37eeab3-7312-4381-b0cb-55968b82c051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:00.715116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.715016    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5ls2\" (UniqueName: \"kubernetes.io/projected/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-kube-api-access-h5ls2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j\" (UID: \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:00.720115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.716620    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr67t\" (UniqueName: \"kubernetes.io/projected/dc483cfa-c134-4176-b6b3-71af38093a2f-kube-api-access-kr67t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz\" (UID: \"dc483cfa-c134-4176-b6b3-71af38093a2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:00.766565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.762426    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:00.827443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.819715    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40530b75-ac11-4d5c-ad11-3e4f5726d972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5\" (UID: \"40530b75-ac11-4d5c-ad11-3e4f5726d972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:00.827443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.819789    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d96f53-8b5e-4f32-adba-ac197a54931f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc\" (UID: \"29d96f53-8b5e-4f32-adba-ac197a54931f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:00.827443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.819865    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgrd6\" (UniqueName: \"kubernetes.io/projected/40530b75-ac11-4d5c-ad11-3e4f5726d972-kube-api-access-hgrd6\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5\" (UID: \"40530b75-ac11-4d5c-ad11-3e4f5726d972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:00.827443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.819942    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bq9\" (UniqueName: \"kubernetes.io/projected/29d96f53-8b5e-4f32-adba-ac197a54931f-kube-api-access-v8bq9\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc\" (UID: \"29d96f53-8b5e-4f32-adba-ac197a54931f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:00.827443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.820074    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e42c7-d074-49ed-9355-b9cd77d0aace-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j\" (UID: \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:00.827443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.820162    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hh9\" (UniqueName: \"kubernetes.io/projected/7e8e42c7-d074-49ed-9355-b9cd77d0aace-kube-api-access-24hh9\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j\" (UID: \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:00.842596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.842530    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:00.843283 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.843231    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e42c7-d074-49ed-9355-b9cd77d0aace-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j\" (UID: \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:00.844089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.843918    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40530b75-ac11-4d5c-ad11-3e4f5726d972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5\" (UID: \"40530b75-ac11-4d5c-ad11-3e4f5726d972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:00.846171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.844502    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d96f53-8b5e-4f32-adba-ac197a54931f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc\" (UID: \"29d96f53-8b5e-4f32-adba-ac197a54931f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:00.865095 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.862594    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgrd6\" (UniqueName: \"kubernetes.io/projected/40530b75-ac11-4d5c-ad11-3e4f5726d972-kube-api-access-hgrd6\") pod \"maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5\" (UID: \"40530b75-ac11-4d5c-ad11-3e4f5726d972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:00.865931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.865896    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bq9\" (UniqueName: \"kubernetes.io/projected/29d96f53-8b5e-4f32-adba-ac197a54931f-kube-api-access-v8bq9\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc\" (UID: \"29d96f53-8b5e-4f32-adba-ac197a54931f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:00.880883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.871700    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"]
Oct 31 02:15:00.880883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.877539    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:00.880883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.879033    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"]
Oct 31 02:15:00.881444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.881385    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hh9\" (UniqueName: \"kubernetes.io/projected/7e8e42c7-d074-49ed-9355-b9cd77d0aace-kube-api-access-24hh9\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j\" (UID: \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:00.886684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.886663    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:00.902708 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.902682    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:00.908453 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.908429    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:00.924348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.922144    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:00.924348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.922300    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:15:00.929432 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.928465    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:00.983069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:00.982795    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:01.002292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.001986    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:01.010125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.009963    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:01.029321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.027153    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gvv\" (UniqueName: \"kubernetes.io/projected/3f485c7f-7eb0-44ba-976f-13d6b19d9101-kube-api-access-w7gvv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.029321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.027235    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.029321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.027438    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.134587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.132693    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gvv\" (UniqueName: \"kubernetes.io/projected/3f485c7f-7eb0-44ba-976f-13d6b19d9101-kube-api-access-w7gvv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.134587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.132769    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.134587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.132932    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.134587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.134021    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.168102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.164108    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.168102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.165970    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gvv\" (UniqueName: \"kubernetes.io/projected/3f485c7f-7eb0-44ba-976f-13d6b19d9101-kube-api-access-w7gvv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.177344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.172347    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"]
Oct 31 02:15:01.177344 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:01.173752    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117b4c77_4f1d_4b01_8665_f641b77b757c.slice/crio-fbc5c2482181351f1fed401779dbd0c10bfe22dc1936068e7dbed178d2925474 WatchSource:0}: Error finding container fbc5c2482181351f1fed401779dbd0c10bfe22dc1936068e7dbed178d2925474: Status 404 returned error can't find the container with id fbc5c2482181351f1fed401779dbd0c10bfe22dc1936068e7dbed178d2925474
Oct 31 02:15:01.239343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.239202    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:01.331664 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.331633    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn" podUID="43e2b864-2259-4446-89ad-0052cc9519b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:15:01.420883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.420860    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"]
Oct 31 02:15:01.423447 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:01.423413    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a60988_774c_48a9_9a51_437c0e70c633.slice/crio-e3b456d004db087159c48abe951baddf5dce9862e19d50f917e08f9515be5527 WatchSource:0}: Error finding container e3b456d004db087159c48abe951baddf5dce9862e19d50f917e08f9515be5527: Status 404 returned error can't find the container with id e3b456d004db087159c48abe951baddf5dce9862e19d50f917e08f9515be5527
Oct 31 02:15:01.473468 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.473444    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"]
Oct 31 02:15:01.475394 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:01.475351    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37eeab3_7312_4381_b0cb_55968b82c051.slice/crio-f79103472edbd6f8ae46daaf89f6ea188e797916c9788b9e13001d158fb5a8cc WatchSource:0}: Error finding container f79103472edbd6f8ae46daaf89f6ea188e797916c9788b9e13001d158fb5a8cc: Status 404 returned error can't find the container with id f79103472edbd6f8ae46daaf89f6ea188e797916c9788b9e13001d158fb5a8cc
Oct 31 02:15:01.485971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.485949    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"]
Oct 31 02:15:01.490307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.490282    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"]
Oct 31 02:15:01.491172 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:01.491144    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40530b75_ac11_4d5c_ad11_3e4f5726d972.slice/crio-c23e42787c8c3b8ca97ed4f9e11fc7ddf4b7977d0162bba4c35743073bf1be74 WatchSource:0}: Error finding container c23e42787c8c3b8ca97ed4f9e11fc7ddf4b7977d0162bba4c35743073bf1be74: Status 404 returned error can't find the container with id c23e42787c8c3b8ca97ed4f9e11fc7ddf4b7977d0162bba4c35743073bf1be74
Oct 31 02:15:01.493725 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:01.493700    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b53e41f_48b3_4e5a_b600_8bca52c3526c.slice/crio-df98ffe6a3f40483c26de1de26ccf07b9b5e354a3c4528cad72a4dae09e46136 WatchSource:0}: Error finding container df98ffe6a3f40483c26de1de26ccf07b9b5e354a3c4528cad72a4dae09e46136: Status 404 returned error can't find the container with id df98ffe6a3f40483c26de1de26ccf07b9b5e354a3c4528cad72a4dae09e46136
Oct 31 02:15:01.536557 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.536524    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"]
Oct 31 02:15:01.548404 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.544416    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"]
Oct 31 02:15:01.550516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.550488    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"]
Oct 31 02:15:01.609957 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.609935    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"]
Oct 31 02:15:01.615183 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.615159    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"]
Oct 31 02:15:01.688441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.686114    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"]
Oct 31 02:15:01.720580 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:01.720527    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc483cfa_c134_4176_b6b3_71af38093a2f.slice/crio-d6f0f520b7d94c88095f4f530903a4f5d9e20161529c1b0bb044273008312106 WatchSource:0}: Error finding container d6f0f520b7d94c88095f4f530903a4f5d9e20161529c1b0bb044273008312106: Status 404 returned error can't find the container with id d6f0f520b7d94c88095f4f530903a4f5d9e20161529c1b0bb044273008312106
Oct 31 02:15:01.721329 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:01.721165    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575a3759_7be6_416a_97bc_d2d5e6e96685.slice/crio-e4171dc52a09b4aa475b85f7a04bd02f8dcb8c1f2a656643bbae7b9ce186b6b4 WatchSource:0}: Error finding container e4171dc52a09b4aa475b85f7a04bd02f8dcb8c1f2a656643bbae7b9ce186b6b4: Status 404 returned error can't find the container with id e4171dc52a09b4aa475b85f7a04bd02f8dcb8c1f2a656643bbae7b9ce186b6b4
Oct 31 02:15:01.721596 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:01.721575    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9990e4_da2e_4897_a3c1_4a9e9d7271b4.slice/crio-b15b906bfc9e0919b1c4f3783bf625c88892d187f5cdd3d6571a94d0dbdd617b WatchSource:0}: Error finding container b15b906bfc9e0919b1c4f3783bf625c88892d187f5cdd3d6571a94d0dbdd617b: Status 404 returned error can't find the container with id b15b906bfc9e0919b1c4f3783bf625c88892d187f5cdd3d6571a94d0dbdd617b
Oct 31 02:15:01.728887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.726736    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" event={"ID":"9b53e41f-48b3-4e5a-b600-8bca52c3526c","Type":"ContainerStarted","Data":"df98ffe6a3f40483c26de1de26ccf07b9b5e354a3c4528cad72a4dae09e46136"}
Oct 31 02:15:01.730450 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.729996    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" event={"ID":"117b4c77-4f1d-4b01-8665-f641b77b757c","Type":"ContainerStarted","Data":"fbc5c2482181351f1fed401779dbd0c10bfe22dc1936068e7dbed178d2925474"}
Oct 31 02:15:01.732307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.732250    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" event={"ID":"24a60988-774c-48a9-9a51-437c0e70c633","Type":"ContainerStarted","Data":"e3b456d004db087159c48abe951baddf5dce9862e19d50f917e08f9515be5527"}
Oct 31 02:15:01.734209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.734146    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" event={"ID":"29d96f53-8b5e-4f32-adba-ac197a54931f","Type":"ContainerStarted","Data":"1d7803c8129017911e83ac837e90e5078517fe958dcc74b1d55486a9344fe147"}
Oct 31 02:15:01.742113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.741845    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" event={"ID":"40530b75-ac11-4d5c-ad11-3e4f5726d972","Type":"ContainerStarted","Data":"c23e42787c8c3b8ca97ed4f9e11fc7ddf4b7977d0162bba4c35743073bf1be74"}
Oct 31 02:15:01.748145 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.746402    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" event={"ID":"a37eeab3-7312-4381-b0cb-55968b82c051","Type":"ContainerStarted","Data":"f79103472edbd6f8ae46daaf89f6ea188e797916c9788b9e13001d158fb5a8cc"}
Oct 31 02:15:01.752182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.751388    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm" event={"ID":"3f485c7f-7eb0-44ba-976f-13d6b19d9101","Type":"ContainerStarted","Data":"d3b219d8ff2a0188d2adca563d103eee4a96a9fc02b301fbb2ccc1d5f897ab7f"}
Oct 31 02:15:01.763866 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:01.763189    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" event={"ID":"7e8e42c7-d074-49ed-9355-b9cd77d0aace","Type":"ContainerStarted","Data":"1faa7d61b91f1e996ea68d9185ec98e75602f2f8756d11d32a43fc733bb4b06c"}
Oct 31 02:15:02.776295 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.776270    5293 generic.go:356] "Generic (PLEG): container finished" podID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerID="72fc6eff0f4f755ba46fa259b99c615d0d7b5af872657e9f0f75a1e4c53f4d3e" exitCode=60
Oct 31 02:15:02.776637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.776353    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" event={"ID":"9b53e41f-48b3-4e5a-b600-8bca52c3526c","Type":"ContainerDied","Data":"72fc6eff0f4f755ba46fa259b99c615d0d7b5af872657e9f0f75a1e4c53f4d3e"}
Oct 31 02:15:02.776689 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.776651    5293 scope.go:122] "RemoveContainer" containerID="72fc6eff0f4f755ba46fa259b99c615d0d7b5af872657e9f0f75a1e4c53f4d3e"
Oct 31 02:15:02.778023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.778005    5293 generic.go:356] "Generic (PLEG): container finished" podID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerID="218ae26591aeb811e8104cc7122ecee9ca15d7fce39a12ed7f45be20f4ade874" exitCode=60
Oct 31 02:15:02.778138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.778098    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" event={"ID":"117b4c77-4f1d-4b01-8665-f641b77b757c","Type":"ContainerDied","Data":"218ae26591aeb811e8104cc7122ecee9ca15d7fce39a12ed7f45be20f4ade874"}
Oct 31 02:15:02.778400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.778381    5293 scope.go:122] "RemoveContainer" containerID="218ae26591aeb811e8104cc7122ecee9ca15d7fce39a12ed7f45be20f4ade874"
Oct 31 02:15:02.779849 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.779818    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" event={"ID":"24a60988-774c-48a9-9a51-437c0e70c633","Type":"ContainerStarted","Data":"c85ee3281d8f6b66b5c27793f6187edd7c3fa998e14e6d4dd99463faa8c1f10b"}
Oct 31 02:15:02.781571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.781552    5293 generic.go:356] "Generic (PLEG): container finished" podID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerID="1a3f185ce518f2ff493066179f12911c20983620298f9e7bd364f6fa7f3d31aa" exitCode=60
Oct 31 02:15:02.781665 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.781643    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" event={"ID":"29d96f53-8b5e-4f32-adba-ac197a54931f","Type":"ContainerDied","Data":"1a3f185ce518f2ff493066179f12911c20983620298f9e7bd364f6fa7f3d31aa"}
Oct 31 02:15:02.781818 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.781795    5293 scope.go:122] "RemoveContainer" containerID="1a3f185ce518f2ff493066179f12911c20983620298f9e7bd364f6fa7f3d31aa"
Oct 31 02:15:02.783275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.783252    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" event={"ID":"575a3759-7be6-416a-97bc-d2d5e6e96685","Type":"ContainerStarted","Data":"b6c0062c77c6ab98bddfa89c364cc2f5dfca8bf0ecae69e50b583bdf7c4cf682"}
Oct 31 02:15:02.783368 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.783292    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" event={"ID":"575a3759-7be6-416a-97bc-d2d5e6e96685","Type":"ContainerStarted","Data":"e4171dc52a09b4aa475b85f7a04bd02f8dcb8c1f2a656643bbae7b9ce186b6b4"}
Oct 31 02:15:02.785997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.785975    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" event={"ID":"40530b75-ac11-4d5c-ad11-3e4f5726d972","Type":"ContainerStarted","Data":"12fd1fd4dbb0e53d3d6adee8ccda3426faa071326694d59a7f20d41f7f5d838e"}
Oct 31 02:15:02.787653 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.787633    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" event={"ID":"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4","Type":"ContainerStarted","Data":"3ec290798e9e844a37605d7c6a8da6a5c91fbd891490820e921dd9082fbb51b2"}
Oct 31 02:15:02.787755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.787670    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" event={"ID":"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4","Type":"ContainerStarted","Data":"b15b906bfc9e0919b1c4f3783bf625c88892d187f5cdd3d6571a94d0dbdd617b"}
Oct 31 02:15:02.789404 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.789378    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" event={"ID":"a37eeab3-7312-4381-b0cb-55968b82c051","Type":"ContainerStarted","Data":"773e98d18b20b8c0dd981ed0d332389b04bd6f68b1442c4df665eb0831c810a4"}
Oct 31 02:15:02.791208 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.791186    5293 generic.go:356] "Generic (PLEG): container finished" podID="3f485c7f-7eb0-44ba-976f-13d6b19d9101" containerID="7be9f6a668d5ba9c383a73c1ac0142fa69764ef0ff39c919874490d250d094e9" exitCode=0
Oct 31 02:15:02.791307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.791291    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm" event={"ID":"3f485c7f-7eb0-44ba-976f-13d6b19d9101","Type":"ContainerDied","Data":"7be9f6a668d5ba9c383a73c1ac0142fa69764ef0ff39c919874490d250d094e9"}
Oct 31 02:15:02.793712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.793689    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" event={"ID":"dc483cfa-c134-4176-b6b3-71af38093a2f","Type":"ContainerStarted","Data":"3fbb69bdd048e553533b641996d35033b6b27040276c0c4dec4ec56d806caa22"}
Oct 31 02:15:02.793825 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.793733    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" event={"ID":"dc483cfa-c134-4176-b6b3-71af38093a2f","Type":"ContainerStarted","Data":"d6f0f520b7d94c88095f4f530903a4f5d9e20161529c1b0bb044273008312106"}
Oct 31 02:15:02.795896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.795872    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" event={"ID":"7e8e42c7-d074-49ed-9355-b9cd77d0aace","Type":"ContainerStarted","Data":"a1b33f5e765db7460360d9f56fde64c13e44a8e9a22c72065622ed7063f02668"}
Oct 31 02:15:02.855996 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.855937    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" podStartSLOduration=2.855915982 podStartE2EDuration="2.855915982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.852538548 +0000 UTC m=+3260.801186941" watchObservedRunningTime="2026-10-31 02:15:02.855915982 +0000 UTC m=+3260.804564376"
Oct 31 02:15:02.893465 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.893419    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" podStartSLOduration=2.893405281 podStartE2EDuration="2.893405281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.892277934 +0000 UTC m=+3260.840926326" watchObservedRunningTime="2026-10-31 02:15:02.893405281 +0000 UTC m=+3260.842053673"
Oct 31 02:15:02.912405 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.912348    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" podStartSLOduration=2.912328896 podStartE2EDuration="2.912328896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.910408394 +0000 UTC m=+3260.859056785" watchObservedRunningTime="2026-10-31 02:15:02.912328896 +0000 UTC m=+3260.860977286"
Oct 31 02:15:02.929436 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.929391    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" podStartSLOduration=2.929377856 podStartE2EDuration="2.929377856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.926219548 +0000 UTC m=+3260.874867954" watchObservedRunningTime="2026-10-31 02:15:02.929377856 +0000 UTC m=+3260.878026247"
Oct 31 02:15:02.943851 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.942468    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" podStartSLOduration=2.942453114 podStartE2EDuration="2.942453114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.941535569 +0000 UTC m=+3260.890183963" watchObservedRunningTime="2026-10-31 02:15:02.942453114 +0000 UTC m=+3260.891101508"
Oct 31 02:15:02.959061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.958957    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" podStartSLOduration=2.9589453 podStartE2EDuration="2.9589453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.957097797 +0000 UTC m=+3260.905746189" watchObservedRunningTime="2026-10-31 02:15:02.9589453 +0000 UTC m=+3260.907593692"
Oct 31 02:15:02.974944 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:02.974893    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" podStartSLOduration=2.974876412 podStartE2EDuration="2.974876412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.973063605 +0000 UTC m=+3260.921711992" watchObservedRunningTime="2026-10-31 02:15:02.974876412 +0000 UTC m=+3260.923524804"
Oct 31 02:15:03.846690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.839315    5293 generic.go:356] "Generic (PLEG): container finished" podID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerID="a1b33f5e765db7460360d9f56fde64c13e44a8e9a22c72065622ed7063f02668" exitCode=60
Oct 31 02:15:03.846690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.839482    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" event={"ID":"7e8e42c7-d074-49ed-9355-b9cd77d0aace","Type":"ContainerDied","Data":"a1b33f5e765db7460360d9f56fde64c13e44a8e9a22c72065622ed7063f02668"}
Oct 31 02:15:03.846690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.840055    5293 scope.go:122] "RemoveContainer" containerID="a1b33f5e765db7460360d9f56fde64c13e44a8e9a22c72065622ed7063f02668"
Oct 31 02:15:03.868080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.868033    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" event={"ID":"9b53e41f-48b3-4e5a-b600-8bca52c3526c","Type":"ContainerStarted","Data":"4f8f4b030b401e0c06702ee65d32dd6126a0c6f513588f0706d0d97051632200"}
Oct 31 02:15:03.876680 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.876651    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" event={"ID":"117b4c77-4f1d-4b01-8665-f641b77b757c","Type":"ContainerStarted","Data":"501af2e4aaa86c5bc4ace3639f46bc74aab229cc445f1b8a5f56e43034cb3f9d"}
Oct 31 02:15:03.880643 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.880614    5293 generic.go:356] "Generic (PLEG): container finished" podID="24a60988-774c-48a9-9a51-437c0e70c633" containerID="c85ee3281d8f6b66b5c27793f6187edd7c3fa998e14e6d4dd99463faa8c1f10b" exitCode=60
Oct 31 02:15:03.880745 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.880711    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" event={"ID":"24a60988-774c-48a9-9a51-437c0e70c633","Type":"ContainerDied","Data":"c85ee3281d8f6b66b5c27793f6187edd7c3fa998e14e6d4dd99463faa8c1f10b"}
Oct 31 02:15:03.880920 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.880900    5293 scope.go:122] "RemoveContainer" containerID="c85ee3281d8f6b66b5c27793f6187edd7c3fa998e14e6d4dd99463faa8c1f10b"
Oct 31 02:15:03.884476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.884450    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" event={"ID":"29d96f53-8b5e-4f32-adba-ac197a54931f","Type":"ContainerStarted","Data":"65b51f3d62f622bfcde8cb98cf17c8167c0ee6f164572983a0ce3f6715aee0b1"}
Oct 31 02:15:03.894489 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.894467    5293 generic.go:356] "Generic (PLEG): container finished" podID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerID="b6c0062c77c6ab98bddfa89c364cc2f5dfca8bf0ecae69e50b583bdf7c4cf682" exitCode=60
Oct 31 02:15:03.894567 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.894543    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" event={"ID":"575a3759-7be6-416a-97bc-d2d5e6e96685","Type":"ContainerDied","Data":"b6c0062c77c6ab98bddfa89c364cc2f5dfca8bf0ecae69e50b583bdf7c4cf682"}
Oct 31 02:15:03.894798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.894782    5293 scope.go:122] "RemoveContainer" containerID="b6c0062c77c6ab98bddfa89c364cc2f5dfca8bf0ecae69e50b583bdf7c4cf682"
Oct 31 02:15:03.898357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.898337    5293 generic.go:356] "Generic (PLEG): container finished" podID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerID="12fd1fd4dbb0e53d3d6adee8ccda3426faa071326694d59a7f20d41f7f5d838e" exitCode=60
Oct 31 02:15:03.899148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.898507    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" event={"ID":"40530b75-ac11-4d5c-ad11-3e4f5726d972","Type":"ContainerDied","Data":"12fd1fd4dbb0e53d3d6adee8ccda3426faa071326694d59a7f20d41f7f5d838e"}
Oct 31 02:15:03.899258 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.898991    5293 scope.go:122] "RemoveContainer" containerID="12fd1fd4dbb0e53d3d6adee8ccda3426faa071326694d59a7f20d41f7f5d838e"
Oct 31 02:15:03.903979 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.903954    5293 generic.go:356] "Generic (PLEG): container finished" podID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerID="3ec290798e9e844a37605d7c6a8da6a5c91fbd891490820e921dd9082fbb51b2" exitCode=60
Oct 31 02:15:03.904102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.904068    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" event={"ID":"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4","Type":"ContainerDied","Data":"3ec290798e9e844a37605d7c6a8da6a5c91fbd891490820e921dd9082fbb51b2"}
Oct 31 02:15:03.904423 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.904402    5293 scope.go:122] "RemoveContainer" containerID="3ec290798e9e844a37605d7c6a8da6a5c91fbd891490820e921dd9082fbb51b2"
Oct 31 02:15:03.907961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.907941    5293 generic.go:356] "Generic (PLEG): container finished" podID="a37eeab3-7312-4381-b0cb-55968b82c051" containerID="773e98d18b20b8c0dd981ed0d332389b04bd6f68b1442c4df665eb0831c810a4" exitCode=60
Oct 31 02:15:03.908349 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.908031    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" podStartSLOduration=3.9080158750000002 podStartE2EDuration="3.908015875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:03.900247012 +0000 UTC m=+3261.848895405" watchObservedRunningTime="2026-10-31 02:15:03.908015875 +0000 UTC m=+3261.856664267"
Oct 31 02:15:03.908553 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.908193    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" event={"ID":"a37eeab3-7312-4381-b0cb-55968b82c051","Type":"ContainerDied","Data":"773e98d18b20b8c0dd981ed0d332389b04bd6f68b1442c4df665eb0831c810a4"}
Oct 31 02:15:03.909720 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.908377    5293 scope.go:122] "RemoveContainer" containerID="773e98d18b20b8c0dd981ed0d332389b04bd6f68b1442c4df665eb0831c810a4"
Oct 31 02:15:03.915158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.915062    5293 generic.go:356] "Generic (PLEG): container finished" podID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerID="3fbb69bdd048e553533b641996d35033b6b27040276c0c4dec4ec56d806caa22" exitCode=60
Oct 31 02:15:03.915529 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.915510    5293 scope.go:122] "RemoveContainer" containerID="3fbb69bdd048e553533b641996d35033b6b27040276c0c4dec4ec56d806caa22"
Oct 31 02:15:03.915851 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.915831    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" event={"ID":"dc483cfa-c134-4176-b6b3-71af38093a2f","Type":"ContainerDied","Data":"3fbb69bdd048e553533b641996d35033b6b27040276c0c4dec4ec56d806caa22"}
Oct 31 02:15:03.944702 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:03.944643    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" podStartSLOduration=3.944625121 podStartE2EDuration="3.944625121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:03.924068569 +0000 UTC m=+3261.872716958" watchObservedRunningTime="2026-10-31 02:15:03.944625121 +0000 UTC m=+3261.893273515"
Oct 31 02:15:04.069699 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.069668    5293 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:15:04.069797 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.069727    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:15:04.078095 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.078025    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" podStartSLOduration=4.078008502 podStartE2EDuration="4.078008502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:04.072644816 +0000 UTC m=+3262.021293205" watchObservedRunningTime="2026-10-31 02:15:04.078008502 +0000 UTC m=+3262.026656897"
Oct 31 02:15:04.466747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.466678    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:04.579258 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.579233    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-openshift-ca\") pod \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") "
Oct 31 02:15:04.579465 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.579449    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-xks-ca\") pod \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") "
Oct 31 02:15:04.579519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.579502    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f485c7f-7eb0-44ba-976f-13d6b19d9101-kube-api-access-w7gvv\" (UniqueName: \"kubernetes.io/projected/3f485c7f-7eb0-44ba-976f-13d6b19d9101-kube-api-access-w7gvv\") pod \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\" (UID: \"3f485c7f-7eb0-44ba-976f-13d6b19d9101\") "
Oct 31 02:15:04.579581 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.579556    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-openshift-ca" pod "3f485c7f-7eb0-44ba-976f-13d6b19d9101" (UID: "3f485c7f-7eb0-44ba-976f-13d6b19d9101"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:04.580093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.580019    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:04.599610 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.598815    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-xks-ca" pod "3f485c7f-7eb0-44ba-976f-13d6b19d9101" (UID: "3f485c7f-7eb0-44ba-976f-13d6b19d9101"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:15:04.600248 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.600216    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f485c7f-7eb0-44ba-976f-13d6b19d9101-kube-api-access-w7gvv" pod "3f485c7f-7eb0-44ba-976f-13d6b19d9101" (UID: "3f485c7f-7eb0-44ba-976f-13d6b19d9101"). InnerVolumeSpecName "kube-api-access-w7gvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:04.682245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.682204    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f485c7f-7eb0-44ba-976f-13d6b19d9101-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:04.682245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.682239    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7gvv\" (UniqueName: \"kubernetes.io/projected/3f485c7f-7eb0-44ba-976f-13d6b19d9101-kube-api-access-w7gvv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:04.926549 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.926512    5293 generic.go:356] "Generic (PLEG): container finished" podID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerID="65b51f3d62f622bfcde8cb98cf17c8167c0ee6f164572983a0ce3f6715aee0b1" exitCode=60
Oct 31 02:15:04.927018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.926644    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" event={"ID":"29d96f53-8b5e-4f32-adba-ac197a54931f","Type":"ContainerDied","Data":"65b51f3d62f622bfcde8cb98cf17c8167c0ee6f164572983a0ce3f6715aee0b1"}
Oct 31 02:15:04.927018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.926710    5293 scope.go:122] "RemoveContainer" containerID="1a3f185ce518f2ff493066179f12911c20983620298f9e7bd364f6fa7f3d31aa"
Oct 31 02:15:04.927018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.926986    5293 scope.go:122] "RemoveContainer" containerID="65b51f3d62f622bfcde8cb98cf17c8167c0ee6f164572983a0ce3f6715aee0b1"
Oct 31 02:15:04.927473 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:04.927446    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc_odh-ai-gateway-infra(29d96f53-8b5e-4f32-adba-ac197a54931f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f"
Oct 31 02:15:04.929187 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.929163    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" event={"ID":"575a3759-7be6-416a-97bc-d2d5e6e96685","Type":"ContainerStarted","Data":"c98cde77616fd943a6f6a6375cdfec0a372fd2e463f3b37a1317463a3e7c39fe"}
Oct 31 02:15:04.931781 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.931757    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" event={"ID":"40530b75-ac11-4d5c-ad11-3e4f5726d972","Type":"ContainerStarted","Data":"e2705036f553f43c64dba4caf42dae75b6b1d9e6349430654e7a4b8e236caac1"}
Oct 31 02:15:04.934073 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.934051    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" event={"ID":"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4","Type":"ContainerStarted","Data":"6991be3c407bee8aa9d63bd7c356baee0d55eb011c137dfeb2d4f68688a397ad"}
Oct 31 02:15:04.936435 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.936407    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" event={"ID":"a37eeab3-7312-4381-b0cb-55968b82c051","Type":"ContainerStarted","Data":"b6049308365d92e7c1a8e992b28e50f8c9d79909ea10286920d5896b93863464"}
Oct 31 02:15:04.937772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.937755    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"
Oct 31 02:15:04.937772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.937751    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm" event={"ID":"3f485c7f-7eb0-44ba-976f-13d6b19d9101","Type":"ContainerDied","Data":"d3b219d8ff2a0188d2adca563d103eee4a96a9fc02b301fbb2ccc1d5f897ab7f"}
Oct 31 02:15:04.937941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.937811    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b219d8ff2a0188d2adca563d103eee4a96a9fc02b301fbb2ccc1d5f897ab7f"
Oct 31 02:15:04.941257 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.940734    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" event={"ID":"dc483cfa-c134-4176-b6b3-71af38093a2f","Type":"ContainerStarted","Data":"916f99c8ca1eefcc686780d90247795133320349d0edf2d7f2d7ffc58cf75747"}
Oct 31 02:15:04.942854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.942827    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" event={"ID":"7e8e42c7-d074-49ed-9355-b9cd77d0aace","Type":"ContainerStarted","Data":"44a87b636cc5f5d2e47b9ed49a34a1f0f26254a94a2f02c315ab05c48c6bc098"}
Oct 31 02:15:04.944971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.944953    5293 generic.go:356] "Generic (PLEG): container finished" podID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerID="4f8f4b030b401e0c06702ee65d32dd6126a0c6f513588f0706d0d97051632200" exitCode=60
Oct 31 02:15:04.945114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.945015    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" event={"ID":"9b53e41f-48b3-4e5a-b600-8bca52c3526c","Type":"ContainerDied","Data":"4f8f4b030b401e0c06702ee65d32dd6126a0c6f513588f0706d0d97051632200"}
Oct 31 02:15:04.945181 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.945113    5293 scope.go:122] "RemoveContainer" containerID="72fc6eff0f4f755ba46fa259b99c615d0d7b5af872657e9f0f75a1e4c53f4d3e"
Oct 31 02:15:04.945317 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.945303    5293 scope.go:122] "RemoveContainer" containerID="4f8f4b030b401e0c06702ee65d32dd6126a0c6f513588f0706d0d97051632200"
Oct 31 02:15:04.945542 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:04.945524    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh_odh-ai-gateway-infra(9b53e41f-48b3-4e5a-b600-8bca52c3526c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c"
Oct 31 02:15:04.947489 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.947471    5293 generic.go:356] "Generic (PLEG): container finished" podID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerID="501af2e4aaa86c5bc4ace3639f46bc74aab229cc445f1b8a5f56e43034cb3f9d" exitCode=60
Oct 31 02:15:04.947595 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.947542    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" event={"ID":"117b4c77-4f1d-4b01-8665-f641b77b757c","Type":"ContainerDied","Data":"501af2e4aaa86c5bc4ace3639f46bc74aab229cc445f1b8a5f56e43034cb3f9d"}
Oct 31 02:15:04.947776 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.947761    5293 scope.go:122] "RemoveContainer" containerID="501af2e4aaa86c5bc4ace3639f46bc74aab229cc445f1b8a5f56e43034cb3f9d"
Oct 31 02:15:04.948024 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:04.948002    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc_odh-ai-gateway-infra(117b4c77-4f1d-4b01-8665-f641b77b757c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c"
Oct 31 02:15:04.952736 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.952712    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" event={"ID":"24a60988-774c-48a9-9a51-437c0e70c633","Type":"ContainerStarted","Data":"79ff53b40b024f83d4b9210db495856309ab73597637766e9151dc44a5c19fbb"}
Oct 31 02:15:04.962769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:04.962747    5293 scope.go:122] "RemoveContainer" containerID="218ae26591aeb811e8104cc7122ecee9ca15d7fce39a12ed7f45be20f4ade874"
Oct 31 02:15:05.964833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.964792    5293 generic.go:356] "Generic (PLEG): container finished" podID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerID="916f99c8ca1eefcc686780d90247795133320349d0edf2d7f2d7ffc58cf75747" exitCode=60
Oct 31 02:15:05.965307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.964879    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" event={"ID":"dc483cfa-c134-4176-b6b3-71af38093a2f","Type":"ContainerDied","Data":"916f99c8ca1eefcc686780d90247795133320349d0edf2d7f2d7ffc58cf75747"}
Oct 31 02:15:05.965307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.964970    5293 scope.go:122] "RemoveContainer" containerID="3fbb69bdd048e553533b641996d35033b6b27040276c0c4dec4ec56d806caa22"
Oct 31 02:15:05.965456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.965407    5293 scope.go:122] "RemoveContainer" containerID="916f99c8ca1eefcc686780d90247795133320349d0edf2d7f2d7ffc58cf75747"
Oct 31 02:15:05.965860 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.965836    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz_odh-ai-gateway-infra(dc483cfa-c134-4176-b6b3-71af38093a2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f"
Oct 31 02:15:05.967720 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.967321    5293 generic.go:356] "Generic (PLEG): container finished" podID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerID="44a87b636cc5f5d2e47b9ed49a34a1f0f26254a94a2f02c315ab05c48c6bc098" exitCode=60
Oct 31 02:15:05.967720 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.967445    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" event={"ID":"7e8e42c7-d074-49ed-9355-b9cd77d0aace","Type":"ContainerDied","Data":"44a87b636cc5f5d2e47b9ed49a34a1f0f26254a94a2f02c315ab05c48c6bc098"}
Oct 31 02:15:05.967881 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.967862    5293 scope.go:122] "RemoveContainer" containerID="44a87b636cc5f5d2e47b9ed49a34a1f0f26254a94a2f02c315ab05c48c6bc098"
Oct 31 02:15:05.968392 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.968367    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j_odh-ai-gateway-infra(7e8e42c7-d074-49ed-9355-b9cd77d0aace)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace"
Oct 31 02:15:05.969959 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.969938    5293 scope.go:122] "RemoveContainer" containerID="4f8f4b030b401e0c06702ee65d32dd6126a0c6f513588f0706d0d97051632200"
Oct 31 02:15:05.970340 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.970316    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh_odh-ai-gateway-infra(9b53e41f-48b3-4e5a-b600-8bca52c3526c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c"
Oct 31 02:15:05.971688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.971670    5293 scope.go:122] "RemoveContainer" containerID="501af2e4aaa86c5bc4ace3639f46bc74aab229cc445f1b8a5f56e43034cb3f9d"
Oct 31 02:15:05.972084 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.972025    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc_odh-ai-gateway-infra(117b4c77-4f1d-4b01-8665-f641b77b757c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c"
Oct 31 02:15:05.973526 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.973506    5293 generic.go:356] "Generic (PLEG): container finished" podID="24a60988-774c-48a9-9a51-437c0e70c633" containerID="79ff53b40b024f83d4b9210db495856309ab73597637766e9151dc44a5c19fbb" exitCode=60
Oct 31 02:15:05.973653 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.973572    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" event={"ID":"24a60988-774c-48a9-9a51-437c0e70c633","Type":"ContainerDied","Data":"79ff53b40b024f83d4b9210db495856309ab73597637766e9151dc44a5c19fbb"}
Oct 31 02:15:05.973862 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.973844    5293 scope.go:122] "RemoveContainer" containerID="79ff53b40b024f83d4b9210db495856309ab73597637766e9151dc44a5c19fbb"
Oct 31 02:15:05.974249 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.974224    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj_odh-ai-gateway-infra(24a60988-774c-48a9-9a51-437c0e70c633)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" podUID="24a60988-774c-48a9-9a51-437c0e70c633"
Oct 31 02:15:05.975693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.975675    5293 scope.go:122] "RemoveContainer" containerID="65b51f3d62f622bfcde8cb98cf17c8167c0ee6f164572983a0ce3f6715aee0b1"
Oct 31 02:15:05.976008 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.975984    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc_odh-ai-gateway-infra(29d96f53-8b5e-4f32-adba-ac197a54931f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f"
Oct 31 02:15:05.977798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.977777    5293 generic.go:356] "Generic (PLEG): container finished" podID="e1321636-259d-41a9-9a08-99daca44e499" containerID="a50fec06a82ce909922661733c8c00dac1ced094df28ea4a51691c862045072e" exitCode=0
Oct 31 02:15:05.977931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.977905    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" event={"ID":"e1321636-259d-41a9-9a08-99daca44e499","Type":"ContainerDied","Data":"a50fec06a82ce909922661733c8c00dac1ced094df28ea4a51691c862045072e"}
Oct 31 02:15:05.980752 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.980731    5293 generic.go:356] "Generic (PLEG): container finished" podID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerID="c98cde77616fd943a6f6a6375cdfec0a372fd2e463f3b37a1317463a3e7c39fe" exitCode=60
Oct 31 02:15:05.980841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.980804    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" event={"ID":"575a3759-7be6-416a-97bc-d2d5e6e96685","Type":"ContainerDied","Data":"c98cde77616fd943a6f6a6375cdfec0a372fd2e463f3b37a1317463a3e7c39fe"}
Oct 31 02:15:05.981158 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.981132    5293 scope.go:122] "RemoveContainer" containerID="c98cde77616fd943a6f6a6375cdfec0a372fd2e463f3b37a1317463a3e7c39fe"
Oct 31 02:15:05.981391 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.981365    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr_odh-ai-gateway-infra(575a3759-7be6-416a-97bc-d2d5e6e96685)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685"
Oct 31 02:15:05.981928 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.981910    5293 scope.go:122] "RemoveContainer" containerID="a1b33f5e765db7460360d9f56fde64c13e44a8e9a22c72065622ed7063f02668"
Oct 31 02:15:05.983093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.983031    5293 generic.go:356] "Generic (PLEG): container finished" podID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerID="e2705036f553f43c64dba4caf42dae75b6b1d9e6349430654e7a4b8e236caac1" exitCode=60
Oct 31 02:15:05.983093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.983086    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" event={"ID":"40530b75-ac11-4d5c-ad11-3e4f5726d972","Type":"ContainerDied","Data":"e2705036f553f43c64dba4caf42dae75b6b1d9e6349430654e7a4b8e236caac1"}
Oct 31 02:15:05.983429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.983412    5293 scope.go:122] "RemoveContainer" containerID="e2705036f553f43c64dba4caf42dae75b6b1d9e6349430654e7a4b8e236caac1"
Oct 31 02:15:05.983734 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.983671    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5_odh-ai-gateway-infra(40530b75-ac11-4d5c-ad11-3e4f5726d972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972"
Oct 31 02:15:05.985283 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.985266    5293 generic.go:356] "Generic (PLEG): container finished" podID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerID="6991be3c407bee8aa9d63bd7c356baee0d55eb011c137dfeb2d4f68688a397ad" exitCode=60
Oct 31 02:15:05.985398 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.985349    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" event={"ID":"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4","Type":"ContainerDied","Data":"6991be3c407bee8aa9d63bd7c356baee0d55eb011c137dfeb2d4f68688a397ad"}
Oct 31 02:15:05.985580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.985563    5293 scope.go:122] "RemoveContainer" containerID="6991be3c407bee8aa9d63bd7c356baee0d55eb011c137dfeb2d4f68688a397ad"
Oct 31 02:15:05.985825 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.985798    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j_odh-ai-gateway-infra(9e9990e4-da2e-4897-a3c1-4a9e9d7271b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4"
Oct 31 02:15:05.987127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.987108    5293 generic.go:356] "Generic (PLEG): container finished" podID="a37eeab3-7312-4381-b0cb-55968b82c051" containerID="b6049308365d92e7c1a8e992b28e50f8c9d79909ea10286920d5896b93863464" exitCode=60
Oct 31 02:15:05.987220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.987136    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" event={"ID":"a37eeab3-7312-4381-b0cb-55968b82c051","Type":"ContainerDied","Data":"b6049308365d92e7c1a8e992b28e50f8c9d79909ea10286920d5896b93863464"}
Oct 31 02:15:05.987420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.987401    5293 scope.go:122] "RemoveContainer" containerID="b6049308365d92e7c1a8e992b28e50f8c9d79909ea10286920d5896b93863464"
Oct 31 02:15:05.987675 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:05.987655    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw_odh-ai-gateway-infra(a37eeab3-7312-4381-b0cb-55968b82c051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" podUID="a37eeab3-7312-4381-b0cb-55968b82c051"
Oct 31 02:15:05.999342 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:05.999320    5293 scope.go:122] "RemoveContainer" containerID="c85ee3281d8f6b66b5c27793f6187edd7c3fa998e14e6d4dd99463faa8c1f10b"
Oct 31 02:15:06.017474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.017453    5293 scope.go:122] "RemoveContainer" containerID="a3a75510fa2a9792efd80a1b67515a5581db9c956c9d0130ff70cbda5e8afcfc"
Oct 31 02:15:06.033319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.033296    5293 scope.go:122] "RemoveContainer" containerID="b6c0062c77c6ab98bddfa89c364cc2f5dfca8bf0ecae69e50b583bdf7c4cf682"
Oct 31 02:15:06.054914 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.054890    5293 scope.go:122] "RemoveContainer" containerID="12fd1fd4dbb0e53d3d6adee8ccda3426faa071326694d59a7f20d41f7f5d838e"
Oct 31 02:15:06.071224 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.071180    5293 scope.go:122] "RemoveContainer" containerID="3ec290798e9e844a37605d7c6a8da6a5c91fbd891490820e921dd9082fbb51b2"
Oct 31 02:15:06.091650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.091624    5293 scope.go:122] "RemoveContainer" containerID="773e98d18b20b8c0dd981ed0d332389b04bd6f68b1442c4df665eb0831c810a4"
Oct 31 02:15:06.389302 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.389273    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 02:15:06.553496 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553410    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-certs\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.553496 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553466    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1321636-259d-41a9-9a08-99daca44e499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1321636-259d-41a9-9a08-99daca44e499-istio-podinfo\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.553740 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553509    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-istio-token\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-istio-token\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.553740 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553607    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-kube-api-access-vg58c\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.553740 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553662    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-credential-socket\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.553740 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553728    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-data\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.553984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553743    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-certs" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.553984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553770    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1321636-259d-41a9-9a08-99daca44e499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1321636-259d-41a9-9a08-99daca44e499-istiod-ca-cert\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.553984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.553928    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-socket\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.554216 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.554004    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-credential-socket" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.554216 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.554085    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-envoy\") pod \"e1321636-259d-41a9-9a08-99daca44e499\" (UID: \"e1321636-259d-41a9-9a08-99daca44e499\") "
Oct 31 02:15:06.554216 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.554131    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-data" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.554388 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.554279    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-socket" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.554388 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.554310    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1321636-259d-41a9-9a08-99daca44e499-istiod-ca-cert" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:06.555184 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.555155    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.555325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.555186    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1321636-259d-41a9-9a08-99daca44e499-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.555325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.555202    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.555325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.555218    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.555325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.555231    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.556563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.556539    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-istio-token" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:06.556722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.556656    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1321636-259d-41a9-9a08-99daca44e499-istio-podinfo" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:15:06.556836 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.556803    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-envoy" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.556924 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.556903    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-kube-api-access-vg58c" pod "e1321636-259d-41a9-9a08-99daca44e499" (UID: "e1321636-259d-41a9-9a08-99daca44e499"). InnerVolumeSpecName "kube-api-access-vg58c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:06.657261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.657230    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1321636-259d-41a9-9a08-99daca44e499-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.657261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.657258    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1321636-259d-41a9-9a08-99daca44e499-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.657261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.657268    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.657505 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.657277    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/e1321636-259d-41a9-9a08-99daca44e499-kube-api-access-vg58c\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.998226 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:06.998193    5293 scope.go:122] "RemoveContainer" containerID="44a87b636cc5f5d2e47b9ed49a34a1f0f26254a94a2f02c315ab05c48c6bc098"
Oct 31 02:15:06.998677 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:06.998612    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j_odh-ai-gateway-infra(7e8e42c7-d074-49ed-9355-b9cd77d0aace)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace"
Oct 31 02:15:07.000191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.000154    5293 scope.go:122] "RemoveContainer" containerID="79ff53b40b024f83d4b9210db495856309ab73597637766e9151dc44a5c19fbb"
Oct 31 02:15:07.000571 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:07.000535    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj_odh-ai-gateway-infra(24a60988-774c-48a9-9a51-437c0e70c633)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" podUID="24a60988-774c-48a9-9a51-437c0e70c633"
Oct 31 02:15:07.001418 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.001397    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"
Oct 31 02:15:07.001526 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.001421    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc" event={"ID":"e1321636-259d-41a9-9a08-99daca44e499","Type":"ContainerDied","Data":"60a4d00c110611040074e3375fe27a01a53c3c21c0c681eaa5ee12f23c3dccd2"}
Oct 31 02:15:07.001591 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.001488    5293 scope.go:122] "RemoveContainer" containerID="a50fec06a82ce909922661733c8c00dac1ced094df28ea4a51691c862045072e"
Oct 31 02:15:07.004093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.004076    5293 scope.go:122] "RemoveContainer" containerID="c98cde77616fd943a6f6a6375cdfec0a372fd2e463f3b37a1317463a3e7c39fe"
Oct 31 02:15:07.004348 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:07.004302    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr_odh-ai-gateway-infra(575a3759-7be6-416a-97bc-d2d5e6e96685)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685"
Oct 31 02:15:07.006165 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.006145    5293 scope.go:122] "RemoveContainer" containerID="e2705036f553f43c64dba4caf42dae75b6b1d9e6349430654e7a4b8e236caac1"
Oct 31 02:15:07.006393 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:07.006375    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5_odh-ai-gateway-infra(40530b75-ac11-4d5c-ad11-3e4f5726d972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972"
Oct 31 02:15:07.008247 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.008228    5293 scope.go:122] "RemoveContainer" containerID="6991be3c407bee8aa9d63bd7c356baee0d55eb011c137dfeb2d4f68688a397ad"
Oct 31 02:15:07.008506 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:07.008483    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j_odh-ai-gateway-infra(9e9990e4-da2e-4897-a3c1-4a9e9d7271b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4"
Oct 31 02:15:07.010284 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.010268    5293 scope.go:122] "RemoveContainer" containerID="b6049308365d92e7c1a8e992b28e50f8c9d79909ea10286920d5896b93863464"
Oct 31 02:15:07.010518 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:07.010493    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw_odh-ai-gateway-infra(a37eeab3-7312-4381-b0cb-55968b82c051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" podUID="a37eeab3-7312-4381-b0cb-55968b82c051"
Oct 31 02:15:07.012577 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.012561    5293 scope.go:122] "RemoveContainer" containerID="916f99c8ca1eefcc686780d90247795133320349d0edf2d7f2d7ffc58cf75747"
Oct 31 02:15:07.012779 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:07.012760    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz_odh-ai-gateway-infra(dc483cfa-c134-4176-b6b3-71af38093a2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f"
Oct 31 02:15:07.176762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.176725    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"]
Oct 31 02:15:07.180511 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:07.180484    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-z46fc"]
Oct 31 02:15:08.176209 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:08.176171    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1321636-259d-41a9-9a08-99daca44e499" path="/var/lib/kubelet/pods/e1321636-259d-41a9-9a08-99daca44e499/volumes"
Oct 31 02:15:11.039367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:11.039332    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"]
Oct 31 02:15:11.385173 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:11.379151    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn" podUID="43e2b864-2259-4446-89ad-0052cc9519b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:15:11.876544 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:11.876509    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:15:15.082918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:15.082880    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:15:15.685345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:15.685321    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 02:15:15.801067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:15.797566    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e2b864-2259-4446-89ad-0052cc9519b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e2b864-2259-4446-89ad-0052cc9519b7-plugins-config-volume\") pod \"43e2b864-2259-4446-89ad-0052cc9519b7\" (UID: \"43e2b864-2259-4446-89ad-0052cc9519b7\") "
Oct 31 02:15:15.801067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:15.797974    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e2b864-2259-4446-89ad-0052cc9519b7-kube-api-access-xx98x\" (UniqueName: \"kubernetes.io/projected/43e2b864-2259-4446-89ad-0052cc9519b7-kube-api-access-xx98x\") pod \"43e2b864-2259-4446-89ad-0052cc9519b7\" (UID: \"43e2b864-2259-4446-89ad-0052cc9519b7\") "
Oct 31 02:15:15.809068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:15.807462    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e2b864-2259-4446-89ad-0052cc9519b7-plugins-config-volume" pod "43e2b864-2259-4446-89ad-0052cc9519b7" (UID: "43e2b864-2259-4446-89ad-0052cc9519b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:15.938126 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:15.937087    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e2b864-2259-4446-89ad-0052cc9519b7-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:15.975907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:15.975851    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e2b864-2259-4446-89ad-0052cc9519b7-kube-api-access-xx98x" pod "43e2b864-2259-4446-89ad-0052cc9519b7" (UID: "43e2b864-2259-4446-89ad-0052cc9519b7"). InnerVolumeSpecName "kube-api-access-xx98x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:16.049066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.045123    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz" podUID="58d4d72e-1061-4b38-bc36-ec9727d25e00" containerName="payload-processing" containerID="cri-o://5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596" gracePeriod=25
Oct 31 02:15:16.049066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.046922    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx98x\" (UniqueName: \"kubernetes.io/projected/43e2b864-2259-4446-89ad-0052cc9519b7-kube-api-access-xx98x\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:16.111507 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.111474    5293 generic.go:356] "Generic (PLEG): container finished" podID="43e2b864-2259-4446-89ad-0052cc9519b7" containerID="92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc" exitCode=0
Oct 31 02:15:16.112197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.112171    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn" event={"ID":"43e2b864-2259-4446-89ad-0052cc9519b7","Type":"ContainerDied","Data":"92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc"}
Oct 31 02:15:16.112353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.112337    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn" event={"ID":"43e2b864-2259-4446-89ad-0052cc9519b7","Type":"ContainerDied","Data":"71b1b01dd144cee5b428209d472123cbe95485ae39f1431a17980c7a6348b5dc"}
Oct 31 02:15:16.112479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.112466    5293 scope.go:122] "RemoveContainer" containerID="92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc"
Oct 31 02:15:16.112735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.112718    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"
Oct 31 02:15:16.156030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.155997    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"]
Oct 31 02:15:16.196228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.179971    5293 scope.go:122] "RemoveContainer" containerID="b6049308365d92e7c1a8e992b28e50f8c9d79909ea10286920d5896b93863464"
Oct 31 02:15:16.196228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.185339    5293 scope.go:122] "RemoveContainer" containerID="4f8f4b030b401e0c06702ee65d32dd6126a0c6f513588f0706d0d97051632200"
Oct 31 02:15:16.196228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.185664    5293 scope.go:122] "RemoveContainer" containerID="6991be3c407bee8aa9d63bd7c356baee0d55eb011c137dfeb2d4f68688a397ad"
Oct 31 02:15:16.196228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.190927    5293 scope.go:122] "RemoveContainer" containerID="501af2e4aaa86c5bc4ace3639f46bc74aab229cc445f1b8a5f56e43034cb3f9d"
Oct 31 02:15:16.196228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.191411    5293 scope.go:122] "RemoveContainer" containerID="e2705036f553f43c64dba4caf42dae75b6b1d9e6349430654e7a4b8e236caac1"
Oct 31 02:15:16.196228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.194431    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0b6abf-6bd8d86887-lbttn"]
Oct 31 02:15:16.476705 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.476684    5293 scope.go:122] "RemoveContainer" containerID="92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc"
Oct 31 02:15:16.479810 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:16.479783    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc\": container with ID starting with 92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc not found: ID does not exist" containerID="92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc"
Oct 31 02:15:16.479937 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.479815    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc"} err="failed to get container status \"92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc\": rpc error: code = NotFound desc = could not find container \"92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc\": container with ID starting with 92bad51456b9caca638754f49b027887176ebbbbb7859504c299be89f13df6bc not found: ID does not exist"
Oct 31 02:15:16.877394 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.877370    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 02:15:16.955848 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.955820    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d4d72e-1061-4b38-bc36-ec9727d25e00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4d72e-1061-4b38-bc36-ec9727d25e00-plugins-config-volume\") pod \"58d4d72e-1061-4b38-bc36-ec9727d25e00\" (UID: \"58d4d72e-1061-4b38-bc36-ec9727d25e00\") "
Oct 31 02:15:16.956051 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.956019    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d4d72e-1061-4b38-bc36-ec9727d25e00-kube-api-access-8gk5z\" (UniqueName: \"kubernetes.io/projected/58d4d72e-1061-4b38-bc36-ec9727d25e00-kube-api-access-8gk5z\") pod \"58d4d72e-1061-4b38-bc36-ec9727d25e00\" (UID: \"58d4d72e-1061-4b38-bc36-ec9727d25e00\") "
Oct 31 02:15:16.956264 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.956244    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d4d72e-1061-4b38-bc36-ec9727d25e00-plugins-config-volume" pod "58d4d72e-1061-4b38-bc36-ec9727d25e00" (UID: "58d4d72e-1061-4b38-bc36-ec9727d25e00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:16.956690 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.956668    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d4d72e-1061-4b38-bc36-ec9727d25e00-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:17.000016 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:16.999735    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d4d72e-1061-4b38-bc36-ec9727d25e00-kube-api-access-8gk5z" pod "58d4d72e-1061-4b38-bc36-ec9727d25e00" (UID: "58d4d72e-1061-4b38-bc36-ec9727d25e00"). InnerVolumeSpecName "kube-api-access-8gk5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:17.058756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.058684    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gk5z\" (UniqueName: \"kubernetes.io/projected/58d4d72e-1061-4b38-bc36-ec9727d25e00-kube-api-access-8gk5z\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:17.123942 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.123905    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" event={"ID":"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4","Type":"ContainerStarted","Data":"ab6c5f2359f27920671e57e0c4d25dd706e012f0873b4010c73b6287b27fe074"}
Oct 31 02:15:17.126381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.126352    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" event={"ID":"a37eeab3-7312-4381-b0cb-55968b82c051","Type":"ContainerStarted","Data":"66c2838e88ac7a4a05bdaa1ee49259ceabc2faed1732e523c65cb638b574268c"}
Oct 31 02:15:17.127815 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.127790    5293 generic.go:356] "Generic (PLEG): container finished" podID="58d4d72e-1061-4b38-bc36-ec9727d25e00" containerID="5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596" exitCode=0
Oct 31 02:15:17.127948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.127848    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"
Oct 31 02:15:17.127948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.127878    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz" event={"ID":"58d4d72e-1061-4b38-bc36-ec9727d25e00","Type":"ContainerDied","Data":"5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596"}
Oct 31 02:15:17.127948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.127923    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz" event={"ID":"58d4d72e-1061-4b38-bc36-ec9727d25e00","Type":"ContainerDied","Data":"372f2f59cd494c463bf436651b0668dd9aad9246b1b78929fd90e2ce02f05ac6"}
Oct 31 02:15:17.127948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.127942    5293 scope.go:122] "RemoveContainer" containerID="5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596"
Oct 31 02:15:17.131399 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.131374    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" event={"ID":"9b53e41f-48b3-4e5a-b600-8bca52c3526c","Type":"ContainerStarted","Data":"19c13292a065fb8e67338204f584b26ececf952646d53e53a95e37e6ede5ef8c"}
Oct 31 02:15:17.133550 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.133527    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" event={"ID":"117b4c77-4f1d-4b01-8665-f641b77b757c","Type":"ContainerStarted","Data":"4155eebcf4b6e419c5cfa583c8fb0925cac79b68eb1ba3cb6b058cb25cb3e74e"}
Oct 31 02:15:17.135673 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.135654    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" event={"ID":"40530b75-ac11-4d5c-ad11-3e4f5726d972","Type":"ContainerStarted","Data":"79e6b0f1820e2f6f750a75c272bcc482d2b0d858a95c362eefc75c91c2c4c93f"}
Oct 31 02:15:17.142231 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.142215    5293 scope.go:122] "RemoveContainer" containerID="5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596"
Oct 31 02:15:17.142520 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:17.142501    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596\": container with ID starting with 5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596 not found: ID does not exist" containerID="5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596"
Oct 31 02:15:17.142582 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.142527    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596"} err="failed to get container status \"5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596\": rpc error: code = NotFound desc = could not find container \"5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596\": container with ID starting with 5c0a720b357630ddb884c65a03525172d32cc913a5b2dfd741d9c06a22a54596 not found: ID does not exist"
Oct 31 02:15:17.166686 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.166666    5293 scope.go:122] "RemoveContainer" containerID="916f99c8ca1eefcc686780d90247795133320349d0edf2d7f2d7ffc58cf75747"
Oct 31 02:15:17.330214 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.330129    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"]
Oct 31 02:15:17.336688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:17.336659    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6d7f5d-55dc497889-qrtcz"]
Oct 31 02:15:18.146668 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.146638    5293 generic.go:356] "Generic (PLEG): container finished" podID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerID="19c13292a065fb8e67338204f584b26ececf952646d53e53a95e37e6ede5ef8c" exitCode=60
Oct 31 02:15:18.147116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.146709    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" event={"ID":"9b53e41f-48b3-4e5a-b600-8bca52c3526c","Type":"ContainerDied","Data":"19c13292a065fb8e67338204f584b26ececf952646d53e53a95e37e6ede5ef8c"}
Oct 31 02:15:18.147116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.146778    5293 scope.go:122] "RemoveContainer" containerID="4f8f4b030b401e0c06702ee65d32dd6126a0c6f513588f0706d0d97051632200"
Oct 31 02:15:18.147315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.147293    5293 scope.go:122] "RemoveContainer" containerID="19c13292a065fb8e67338204f584b26ececf952646d53e53a95e37e6ede5ef8c"
Oct 31 02:15:18.147739 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:18.147700    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh_odh-ai-gateway-infra(9b53e41f-48b3-4e5a-b600-8bca52c3526c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c"
Oct 31 02:15:18.149010 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.148991    5293 generic.go:356] "Generic (PLEG): container finished" podID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerID="4155eebcf4b6e419c5cfa583c8fb0925cac79b68eb1ba3cb6b058cb25cb3e74e" exitCode=60
Oct 31 02:15:18.149144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.149068    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" event={"ID":"117b4c77-4f1d-4b01-8665-f641b77b757c","Type":"ContainerDied","Data":"4155eebcf4b6e419c5cfa583c8fb0925cac79b68eb1ba3cb6b058cb25cb3e74e"}
Oct 31 02:15:18.149446 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.149430    5293 scope.go:122] "RemoveContainer" containerID="4155eebcf4b6e419c5cfa583c8fb0925cac79b68eb1ba3cb6b058cb25cb3e74e"
Oct 31 02:15:18.149835 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:18.149810    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc_odh-ai-gateway-infra(117b4c77-4f1d-4b01-8665-f641b77b757c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c"
Oct 31 02:15:18.151281 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.151263    5293 generic.go:356] "Generic (PLEG): container finished" podID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerID="79e6b0f1820e2f6f750a75c272bcc482d2b0d858a95c362eefc75c91c2c4c93f" exitCode=60
Oct 31 02:15:18.151375 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.151344    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" event={"ID":"40530b75-ac11-4d5c-ad11-3e4f5726d972","Type":"ContainerDied","Data":"79e6b0f1820e2f6f750a75c272bcc482d2b0d858a95c362eefc75c91c2c4c93f"}
Oct 31 02:15:18.151946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.151927    5293 scope.go:122] "RemoveContainer" containerID="79e6b0f1820e2f6f750a75c272bcc482d2b0d858a95c362eefc75c91c2c4c93f"
Oct 31 02:15:18.152301 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:18.152278    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5_odh-ai-gateway-infra(40530b75-ac11-4d5c-ad11-3e4f5726d972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972"
Oct 31 02:15:18.153283 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.153263    5293 generic.go:356] "Generic (PLEG): container finished" podID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerID="ab6c5f2359f27920671e57e0c4d25dd706e012f0873b4010c73b6287b27fe074" exitCode=60
Oct 31 02:15:18.153361 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.153339    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" event={"ID":"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4","Type":"ContainerDied","Data":"ab6c5f2359f27920671e57e0c4d25dd706e012f0873b4010c73b6287b27fe074"}
Oct 31 02:15:18.153703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.153689    5293 scope.go:122] "RemoveContainer" containerID="ab6c5f2359f27920671e57e0c4d25dd706e012f0873b4010c73b6287b27fe074"
Oct 31 02:15:18.153940 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:18.153920    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j_odh-ai-gateway-infra(9e9990e4-da2e-4897-a3c1-4a9e9d7271b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4"
Oct 31 02:15:18.155285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.155268    5293 generic.go:356] "Generic (PLEG): container finished" podID="a37eeab3-7312-4381-b0cb-55968b82c051" containerID="66c2838e88ac7a4a05bdaa1ee49259ceabc2faed1732e523c65cb638b574268c" exitCode=60
Oct 31 02:15:18.155380 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.155334    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" event={"ID":"a37eeab3-7312-4381-b0cb-55968b82c051","Type":"ContainerDied","Data":"66c2838e88ac7a4a05bdaa1ee49259ceabc2faed1732e523c65cb638b574268c"}
Oct 31 02:15:18.155680 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.155661    5293 scope.go:122] "RemoveContainer" containerID="66c2838e88ac7a4a05bdaa1ee49259ceabc2faed1732e523c65cb638b574268c"
Oct 31 02:15:18.155886 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:18.155869    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw_odh-ai-gateway-infra(a37eeab3-7312-4381-b0cb-55968b82c051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" podUID="a37eeab3-7312-4381-b0cb-55968b82c051"
Oct 31 02:15:18.163540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.163512    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" event={"ID":"dc483cfa-c134-4176-b6b3-71af38093a2f","Type":"ContainerStarted","Data":"203830b1e6220d3104fc749fd2cf939fde65e527483946cfdff6e53fc1b89c38"}
Oct 31 02:15:18.164984 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.164967    5293 scope.go:122] "RemoveContainer" containerID="501af2e4aaa86c5bc4ace3639f46bc74aab229cc445f1b8a5f56e43034cb3f9d"
Oct 31 02:15:18.167456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.166476    5293 scope.go:122] "RemoveContainer" containerID="79ff53b40b024f83d4b9210db495856309ab73597637766e9151dc44a5c19fbb"
Oct 31 02:15:18.167623 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.167603    5293 scope.go:122] "RemoveContainer" containerID="65b51f3d62f622bfcde8cb98cf17c8167c0ee6f164572983a0ce3f6715aee0b1"
Oct 31 02:15:18.177943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.177917    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e2b864-2259-4446-89ad-0052cc9519b7" path="/var/lib/kubelet/pods/43e2b864-2259-4446-89ad-0052cc9519b7/volumes"
Oct 31 02:15:18.178470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.178451    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d4d72e-1061-4b38-bc36-ec9727d25e00" path="/var/lib/kubelet/pods/58d4d72e-1061-4b38-bc36-ec9727d25e00/volumes"
Oct 31 02:15:18.187216 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.187184    5293 scope.go:122] "RemoveContainer" containerID="e2705036f553f43c64dba4caf42dae75b6b1d9e6349430654e7a4b8e236caac1"
Oct 31 02:15:18.213758 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.213338    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"]
Oct 31 02:15:18.264738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.264026    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"]
Oct 31 02:15:18.276002 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.275966    5293 scope.go:122] "RemoveContainer" containerID="6991be3c407bee8aa9d63bd7c356baee0d55eb011c137dfeb2d4f68688a397ad"
Oct 31 02:15:18.281027 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.280997    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"]
Oct 31 02:15:18.297137 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.297117    5293 scope.go:122] "RemoveContainer" containerID="b6049308365d92e7c1a8e992b28e50f8c9d79909ea10286920d5896b93863464"
Oct 31 02:15:18.311841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.311814    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"]
Oct 31 02:15:18.320124 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:18.320101    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"]
Oct 31 02:15:19.170545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.170510    5293 scope.go:122] "RemoveContainer" containerID="44a87b636cc5f5d2e47b9ed49a34a1f0f26254a94a2f02c315ab05c48c6bc098"
Oct 31 02:15:19.173376 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.173353    5293 scope.go:122] "RemoveContainer" containerID="c98cde77616fd943a6f6a6375cdfec0a372fd2e463f3b37a1317463a3e7c39fe"
Oct 31 02:15:19.200077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.197865    5293 generic.go:356] "Generic (PLEG): container finished" podID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerID="203830b1e6220d3104fc749fd2cf939fde65e527483946cfdff6e53fc1b89c38" exitCode=60
Oct 31 02:15:19.200077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.198086    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" event={"ID":"dc483cfa-c134-4176-b6b3-71af38093a2f","Type":"ContainerDied","Data":"203830b1e6220d3104fc749fd2cf939fde65e527483946cfdff6e53fc1b89c38"}
Oct 31 02:15:19.200077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.198141    5293 scope.go:122] "RemoveContainer" containerID="916f99c8ca1eefcc686780d90247795133320349d0edf2d7f2d7ffc58cf75747"
Oct 31 02:15:19.200077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.198617    5293 scope.go:122] "RemoveContainer" containerID="203830b1e6220d3104fc749fd2cf939fde65e527483946cfdff6e53fc1b89c38"
Oct 31 02:15:19.200077 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:19.199009    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz_odh-ai-gateway-infra(dc483cfa-c134-4176-b6b3-71af38093a2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f"
Oct 31 02:15:19.214923 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.213989    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" event={"ID":"24a60988-774c-48a9-9a51-437c0e70c633","Type":"ContainerStarted","Data":"7691b461b042eef3da60c7340448b9132601610d529c316b22527a5d50b34738"}
Oct 31 02:15:19.224587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.224557    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" event={"ID":"29d96f53-8b5e-4f32-adba-ac197a54931f","Type":"ContainerStarted","Data":"417dc244270894668de54b5b1bd36cf5c741e5c9ea3736547efbaed4944b0610"}
Oct 31 02:15:19.306463 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.304104    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"]
Oct 31 02:15:19.925699 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.925282    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:19.964362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.945503    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:19.964362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.959313    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-maas-api-service-ca\") pod \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\" (UID: \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\") "
Oct 31 02:15:19.964362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.959379    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-kube-api-access-h5ls2\" (UniqueName: \"kubernetes.io/projected/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-kube-api-access-h5ls2\") pod \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\" (UID: \"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4\") "
Oct 31 02:15:19.964362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.963217    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-kube-api-access-h5ls2" pod "9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" (UID: "9e9990e4-da2e-4897-a3c1-4a9e9d7271b4"). InnerVolumeSpecName "kube-api-access-h5ls2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:19.964362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:19.963524    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-maas-api-service-ca" pod "9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" (UID: "9e9990e4-da2e-4897-a3c1-4a9e9d7271b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:20.006148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.005783    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:20.116100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.113905    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b53e41f-48b3-4e5a-b600-8bca52c3526c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b53e41f-48b3-4e5a-b600-8bca52c3526c-maas-api-service-ca\") pod \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\" (UID: \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\") "
Oct 31 02:15:20.116100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.114057    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37eeab3-7312-4381-b0cb-55968b82c051-kube-api-access-dt8wh\" (UniqueName: \"kubernetes.io/projected/a37eeab3-7312-4381-b0cb-55968b82c051-kube-api-access-dt8wh\") pod \"a37eeab3-7312-4381-b0cb-55968b82c051\" (UID: \"a37eeab3-7312-4381-b0cb-55968b82c051\") "
Oct 31 02:15:20.116100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.114102    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37eeab3-7312-4381-b0cb-55968b82c051-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37eeab3-7312-4381-b0cb-55968b82c051-maas-api-service-ca\") pod \"a37eeab3-7312-4381-b0cb-55968b82c051\" (UID: \"a37eeab3-7312-4381-b0cb-55968b82c051\") "
Oct 31 02:15:20.116100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.114184    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b53e41f-48b3-4e5a-b600-8bca52c3526c-kube-api-access-zx9j9\" (UniqueName: \"kubernetes.io/projected/9b53e41f-48b3-4e5a-b600-8bca52c3526c-kube-api-access-zx9j9\") pod \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\" (UID: \"9b53e41f-48b3-4e5a-b600-8bca52c3526c\") "
Oct 31 02:15:20.116100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.114464    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b53e41f-48b3-4e5a-b600-8bca52c3526c-maas-api-service-ca" pod "9b53e41f-48b3-4e5a-b600-8bca52c3526c" (UID: "9b53e41f-48b3-4e5a-b600-8bca52c3526c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:20.116100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.116011    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.116100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.116053    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5ls2\" (UniqueName: \"kubernetes.io/projected/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4-kube-api-access-h5ls2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.116100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.116070    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b53e41f-48b3-4e5a-b600-8bca52c3526c-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.120695 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.117875    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37eeab3-7312-4381-b0cb-55968b82c051-maas-api-service-ca" pod "a37eeab3-7312-4381-b0cb-55968b82c051" (UID: "a37eeab3-7312-4381-b0cb-55968b82c051"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:20.156401 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.154244    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37eeab3-7312-4381-b0cb-55968b82c051-kube-api-access-dt8wh" pod "a37eeab3-7312-4381-b0cb-55968b82c051" (UID: "a37eeab3-7312-4381-b0cb-55968b82c051"). InnerVolumeSpecName "kube-api-access-dt8wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:20.156667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.156616    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b53e41f-48b3-4e5a-b600-8bca52c3526c-kube-api-access-zx9j9" pod "9b53e41f-48b3-4e5a-b600-8bca52c3526c" (UID: "9b53e41f-48b3-4e5a-b600-8bca52c3526c"). InnerVolumeSpecName "kube-api-access-zx9j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:20.178534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.178505    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:20.186371 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.186300    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:20.218993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.218928    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt8wh\" (UniqueName: \"kubernetes.io/projected/a37eeab3-7312-4381-b0cb-55968b82c051-kube-api-access-dt8wh\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.219160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.219105    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37eeab3-7312-4381-b0cb-55968b82c051-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.219160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.219126    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx9j9\" (UniqueName: \"kubernetes.io/projected/9b53e41f-48b3-4e5a-b600-8bca52c3526c-kube-api-access-zx9j9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.363113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.358945    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"]
Oct 31 02:15:20.366734 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.366682    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117b4c77-4f1d-4b01-8665-f641b77b757c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117b4c77-4f1d-4b01-8665-f641b77b757c-maas-api-service-ca\") pod \"117b4c77-4f1d-4b01-8665-f641b77b757c\" (UID: \"117b4c77-4f1d-4b01-8665-f641b77b757c\") "
Oct 31 02:15:20.367027 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.367006    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40530b75-ac11-4d5c-ad11-3e4f5726d972-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40530b75-ac11-4d5c-ad11-3e4f5726d972-maas-api-service-ca\") pod \"40530b75-ac11-4d5c-ad11-3e4f5726d972\" (UID: \"40530b75-ac11-4d5c-ad11-3e4f5726d972\") "
Oct 31 02:15:20.402674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.402617    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117b4c77-4f1d-4b01-8665-f641b77b757c-maas-api-service-ca" pod "117b4c77-4f1d-4b01-8665-f641b77b757c" (UID: "117b4c77-4f1d-4b01-8665-f641b77b757c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:20.403366 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.403253    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40530b75-ac11-4d5c-ad11-3e4f5726d972-maas-api-service-ca" pod "40530b75-ac11-4d5c-ad11-3e4f5726d972" (UID: "40530b75-ac11-4d5c-ad11-3e4f5726d972"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:20.415093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.367196    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117b4c77-4f1d-4b01-8665-f641b77b757c-kube-api-access-zjw4r\" (UniqueName: \"kubernetes.io/projected/117b4c77-4f1d-4b01-8665-f641b77b757c-kube-api-access-zjw4r\") pod \"117b4c77-4f1d-4b01-8665-f641b77b757c\" (UID: \"117b4c77-4f1d-4b01-8665-f641b77b757c\") "
Oct 31 02:15:20.415093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.405609    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40530b75-ac11-4d5c-ad11-3e4f5726d972-kube-api-access-hgrd6\" (UniqueName: \"kubernetes.io/projected/40530b75-ac11-4d5c-ad11-3e4f5726d972-kube-api-access-hgrd6\") pod \"40530b75-ac11-4d5c-ad11-3e4f5726d972\" (UID: \"40530b75-ac11-4d5c-ad11-3e4f5726d972\") "
Oct 31 02:15:20.415093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.406601    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117b4c77-4f1d-4b01-8665-f641b77b757c-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.415093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.406639    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40530b75-ac11-4d5c-ad11-3e4f5726d972-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.490090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.430415    5293 generic.go:356] "Generic (PLEG): container finished" podID="24a60988-774c-48a9-9a51-437c0e70c633" containerID="7691b461b042eef3da60c7340448b9132601610d529c316b22527a5d50b34738" exitCode=60
Oct 31 02:15:20.490090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.430558    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" event={"ID":"24a60988-774c-48a9-9a51-437c0e70c633","Type":"ContainerDied","Data":"7691b461b042eef3da60c7340448b9132601610d529c316b22527a5d50b34738"}
Oct 31 02:15:20.490090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.430623    5293 scope.go:122] "RemoveContainer" containerID="79ff53b40b024f83d4b9210db495856309ab73597637766e9151dc44a5c19fbb"
Oct 31 02:15:20.544217 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.541580    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117b4c77-4f1d-4b01-8665-f641b77b757c-kube-api-access-zjw4r" pod "117b4c77-4f1d-4b01-8665-f641b77b757c" (UID: "117b4c77-4f1d-4b01-8665-f641b77b757c"). InnerVolumeSpecName "kube-api-access-zjw4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:20.555106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.544770    5293 generic.go:356] "Generic (PLEG): container finished" podID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerID="417dc244270894668de54b5b1bd36cf5c741e5c9ea3736547efbaed4944b0610" exitCode=60
Oct 31 02:15:20.555106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.544952    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" event={"ID":"29d96f53-8b5e-4f32-adba-ac197a54931f","Type":"ContainerDied","Data":"417dc244270894668de54b5b1bd36cf5c741e5c9ea3736547efbaed4944b0610"}
Oct 31 02:15:20.564089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.556206    5293 scope.go:122] "RemoveContainer" containerID="417dc244270894668de54b5b1bd36cf5c741e5c9ea3736547efbaed4944b0610"
Oct 31 02:15:20.564089 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:20.556694    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc_odh-ai-gateway-infra(29d96f53-8b5e-4f32-adba-ac197a54931f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f"
Oct 31 02:15:20.564089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.562227    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40530b75-ac11-4d5c-ad11-3e4f5726d972-kube-api-access-hgrd6" pod "40530b75-ac11-4d5c-ad11-3e4f5726d972" (UID: "40530b75-ac11-4d5c-ad11-3e4f5726d972"). InnerVolumeSpecName "kube-api-access-hgrd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:20.593032 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.577995    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" event={"ID":"575a3759-7be6-416a-97bc-d2d5e6e96685","Type":"ContainerStarted","Data":"b666eeb885efa2fa7be8114c1e2a345f658a5bf5ff650ffa432b81643abefa69"}
Oct 31 02:15:20.621703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.596118    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5" event={"ID":"40530b75-ac11-4d5c-ad11-3e4f5726d972","Type":"ContainerDied","Data":"c23e42787c8c3b8ca97ed4f9e11fc7ddf4b7977d0162bba4c35743073bf1be74"}
Oct 31 02:15:20.621703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.596264    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"
Oct 31 02:15:20.621703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.621012    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjw4r\" (UniqueName: \"kubernetes.io/projected/117b4c77-4f1d-4b01-8665-f641b77b757c-kube-api-access-zjw4r\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.621703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.621068    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgrd6\" (UniqueName: \"kubernetes.io/projected/40530b75-ac11-4d5c-ad11-3e4f5726d972-kube-api-access-hgrd6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.699229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.696739    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j" event={"ID":"9e9990e4-da2e-4897-a3c1-4a9e9d7271b4","Type":"ContainerDied","Data":"b15b906bfc9e0919b1c4f3783bf625c88892d187f5cdd3d6571a94d0dbdd617b"}
Oct 31 02:15:20.699229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.698503    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"
Oct 31 02:15:20.746713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.743990    5293 scope.go:122] "RemoveContainer" containerID="65b51f3d62f622bfcde8cb98cf17c8167c0ee6f164572983a0ce3f6715aee0b1"
Oct 31 02:15:20.746713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.746268    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw" event={"ID":"a37eeab3-7312-4381-b0cb-55968b82c051","Type":"ContainerDied","Data":"f79103472edbd6f8ae46daaf89f6ea188e797916c9788b9e13001d158fb5a8cc"}
Oct 31 02:15:20.746713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.746482    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"
Oct 31 02:15:20.755556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.755524    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"]
Oct 31 02:15:20.774476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.774441    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" event={"ID":"7e8e42c7-d074-49ed-9355-b9cd77d0aace","Type":"ContainerStarted","Data":"f3f9e81097b1e92dae90d93f7d90d5a88e52487227a2fba7ddb4b2631ea1ff6c"}
Oct 31 02:15:20.775204 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.775165    5293 scope.go:122] "RemoveContainer" containerID="f3f9e81097b1e92dae90d93f7d90d5a88e52487227a2fba7ddb4b2631ea1ff6c"
Oct 31 02:15:20.777147 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:20.777021    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j_odh-ai-gateway-infra(7e8e42c7-d074-49ed-9355-b9cd77d0aace)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace"
Oct 31 02:15:20.793928 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.793894    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh" event={"ID":"9b53e41f-48b3-4e5a-b600-8bca52c3526c","Type":"ContainerDied","Data":"df98ffe6a3f40483c26de1de26ccf07b9b5e354a3c4528cad72a4dae09e46136"}
Oct 31 02:15:20.794274 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.794248    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"
Oct 31 02:15:20.796623 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.796594    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc" event={"ID":"117b4c77-4f1d-4b01-8665-f641b77b757c","Type":"ContainerDied","Data":"fbc5c2482181351f1fed401779dbd0c10bfe22dc1936068e7dbed178d2925474"}
Oct 31 02:15:20.796744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.796691    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"
Oct 31 02:15:20.857517 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.857456    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"]
Oct 31 02:15:20.862598 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.862164    5293 scope.go:122] "RemoveContainer" containerID="79e6b0f1820e2f6f750a75c272bcc482d2b0d858a95c362eefc75c91c2c4c93f"
Oct 31 02:15:20.865650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.865627    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890215-gvj9j"]
Oct 31 02:15:20.878721 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.878691    5293 scope.go:122] "RemoveContainer" containerID="ab6c5f2359f27920671e57e0c4d25dd706e012f0873b4010c73b6287b27fe074"
Oct 31 02:15:20.889994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.889963    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"]
Oct 31 02:15:20.921125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.921097    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-20538a-29890215-zdglh"]
Oct 31 02:15:20.929959 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:20.929928    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"]
Oct 31 02:15:21.014863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.012633    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3cf6f4-29890215-4gjw5"]
Oct 31 02:15:21.014863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.012866    5293 scope.go:122] "RemoveContainer" containerID="66c2838e88ac7a4a05bdaa1ee49259ceabc2faed1732e523c65cb638b574268c"
Oct 31 02:15:21.036692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.036640    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"]
Oct 31 02:15:21.046971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.046330    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:21.051245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.051221    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c088fd-29890215-wd7xc"]
Oct 31 02:15:21.062211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.062146    5293 scope.go:122] "RemoveContainer" containerID="19c13292a065fb8e67338204f584b26ececf952646d53e53a95e37e6ede5ef8c"
Oct 31 02:15:21.063484 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.063457    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"]
Oct 31 02:15:21.071680 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.071659    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6d7f5d-29890215-d77vw"]
Oct 31 02:15:21.090141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.086477    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:21.090141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.086767    5293 scope.go:122] "RemoveContainer" containerID="4155eebcf4b6e419c5cfa583c8fb0925cac79b68eb1ba3cb6b058cb25cb3e74e"
Oct 31 02:15:21.113893 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.109980    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc483cfa-c134-4176-b6b3-71af38093a2f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc483cfa-c134-4176-b6b3-71af38093a2f-maas-api-service-ca\") pod \"dc483cfa-c134-4176-b6b3-71af38093a2f\" (UID: \"dc483cfa-c134-4176-b6b3-71af38093a2f\") "
Oct 31 02:15:21.113893 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.110081    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a60988-774c-48a9-9a51-437c0e70c633-kube-api-access-ghj8v\" (UniqueName: \"kubernetes.io/projected/24a60988-774c-48a9-9a51-437c0e70c633-kube-api-access-ghj8v\") pod \"24a60988-774c-48a9-9a51-437c0e70c633\" (UID: \"24a60988-774c-48a9-9a51-437c0e70c633\") "
Oct 31 02:15:21.113893 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.110323    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a60988-774c-48a9-9a51-437c0e70c633-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a60988-774c-48a9-9a51-437c0e70c633-maas-api-service-ca\") pod \"24a60988-774c-48a9-9a51-437c0e70c633\" (UID: \"24a60988-774c-48a9-9a51-437c0e70c633\") "
Oct 31 02:15:21.113893 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.110358    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc483cfa-c134-4176-b6b3-71af38093a2f-kube-api-access-kr67t\" (UniqueName: \"kubernetes.io/projected/dc483cfa-c134-4176-b6b3-71af38093a2f-kube-api-access-kr67t\") pod \"dc483cfa-c134-4176-b6b3-71af38093a2f\" (UID: \"dc483cfa-c134-4176-b6b3-71af38093a2f\") "
Oct 31 02:15:21.116719 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.114296    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc483cfa-c134-4176-b6b3-71af38093a2f-kube-api-access-kr67t" pod "dc483cfa-c134-4176-b6b3-71af38093a2f" (UID: "dc483cfa-c134-4176-b6b3-71af38093a2f"). InnerVolumeSpecName "kube-api-access-kr67t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:21.116719 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.114653    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc483cfa-c134-4176-b6b3-71af38093a2f-maas-api-service-ca" pod "dc483cfa-c134-4176-b6b3-71af38093a2f" (UID: "dc483cfa-c134-4176-b6b3-71af38093a2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:21.117880 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.117855    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a60988-774c-48a9-9a51-437c0e70c633-kube-api-access-ghj8v" pod "24a60988-774c-48a9-9a51-437c0e70c633" (UID: "24a60988-774c-48a9-9a51-437c0e70c633"). InnerVolumeSpecName "kube-api-access-ghj8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:21.118132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.118109    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a60988-774c-48a9-9a51-437c0e70c633-maas-api-service-ca" pod "24a60988-774c-48a9-9a51-437c0e70c633" (UID: "24a60988-774c-48a9-9a51-437c0e70c633"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:21.212674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.212639    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc483cfa-c134-4176-b6b3-71af38093a2f-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.212674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.212674    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghj8v\" (UniqueName: \"kubernetes.io/projected/24a60988-774c-48a9-9a51-437c0e70c633-kube-api-access-ghj8v\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.213136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.212689    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a60988-774c-48a9-9a51-437c0e70c633-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.213136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.212702    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr67t\" (UniqueName: \"kubernetes.io/projected/dc483cfa-c134-4176-b6b3-71af38093a2f-kube-api-access-kr67t\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.721586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.721557    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"]
Oct 31 02:15:21.842544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.842502    5293 generic.go:356] "Generic (PLEG): container finished" podID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerID="f3f9e81097b1e92dae90d93f7d90d5a88e52487227a2fba7ddb4b2631ea1ff6c" exitCode=60
Oct 31 02:15:21.842738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.842659    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" event={"ID":"7e8e42c7-d074-49ed-9355-b9cd77d0aace","Type":"ContainerDied","Data":"f3f9e81097b1e92dae90d93f7d90d5a88e52487227a2fba7ddb4b2631ea1ff6c"}
Oct 31 02:15:21.842738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.842718    5293 scope.go:122] "RemoveContainer" containerID="44a87b636cc5f5d2e47b9ed49a34a1f0f26254a94a2f02c315ab05c48c6bc098"
Oct 31 02:15:21.843242 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.843223    5293 scope.go:122] "RemoveContainer" containerID="f3f9e81097b1e92dae90d93f7d90d5a88e52487227a2fba7ddb4b2631ea1ff6c"
Oct 31 02:15:21.843644 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:21.843615    5293 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j_odh-ai-gateway-infra(7e8e42c7-d074-49ed-9355-b9cd77d0aace)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace"
Oct 31 02:15:21.852926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.852896    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"]
Oct 31 02:15:21.860199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.860170    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj" event={"ID":"24a60988-774c-48a9-9a51-437c0e70c633","Type":"ContainerDied","Data":"e3b456d004db087159c48abe951baddf5dce9862e19d50f917e08f9515be5527"}
Oct 31 02:15:21.860485 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.860463    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"
Oct 31 02:15:21.882644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.882617    5293 generic.go:356] "Generic (PLEG): container finished" podID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerID="b666eeb885efa2fa7be8114c1e2a345f658a5bf5ff650ffa432b81643abefa69" exitCode=60
Oct 31 02:15:21.882915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.882845    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" event={"ID":"575a3759-7be6-416a-97bc-d2d5e6e96685","Type":"ContainerDied","Data":"b666eeb885efa2fa7be8114c1e2a345f658a5bf5ff650ffa432b81643abefa69"}
Oct 31 02:15:21.884533 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.884512    5293 scope.go:122] "RemoveContainer" containerID="7691b461b042eef3da60c7340448b9132601610d529c316b22527a5d50b34738"
Oct 31 02:15:21.892797 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.892778    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz" event={"ID":"dc483cfa-c134-4176-b6b3-71af38093a2f","Type":"ContainerDied","Data":"d6f0f520b7d94c88095f4f530903a4f5d9e20161529c1b0bb044273008312106"}
Oct 31 02:15:21.892997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.892940    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"
Oct 31 02:15:21.904182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.904131    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"]
Oct 31 02:15:21.905857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.905838    5293 scope.go:122] "RemoveContainer" containerID="c98cde77616fd943a6f6a6375cdfec0a372fd2e463f3b37a1317463a3e7c39fe"
Oct 31 02:15:21.912279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.912242    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890215-nbdsj"]
Oct 31 02:15:21.945552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.945526    5293 scope.go:122] "RemoveContainer" containerID="203830b1e6220d3104fc749fd2cf939fde65e527483946cfdff6e53fc1b89c38"
Oct 31 02:15:21.964163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.964132    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"]
Oct 31 02:15:21.969403 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:21.969279    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e57add-29890215-gcggz"]
Oct 31 02:15:22.095429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.095395    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:22.134934 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.134864    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d96f53-8b5e-4f32-adba-ac197a54931f-kube-api-access-v8bq9\" (UniqueName: \"kubernetes.io/projected/29d96f53-8b5e-4f32-adba-ac197a54931f-kube-api-access-v8bq9\") pod \"29d96f53-8b5e-4f32-adba-ac197a54931f\" (UID: \"29d96f53-8b5e-4f32-adba-ac197a54931f\") "
Oct 31 02:15:22.135161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.135140    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d96f53-8b5e-4f32-adba-ac197a54931f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d96f53-8b5e-4f32-adba-ac197a54931f-maas-api-service-ca\") pod \"29d96f53-8b5e-4f32-adba-ac197a54931f\" (UID: \"29d96f53-8b5e-4f32-adba-ac197a54931f\") "
Oct 31 02:15:22.135650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.135607    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d96f53-8b5e-4f32-adba-ac197a54931f-maas-api-service-ca" pod "29d96f53-8b5e-4f32-adba-ac197a54931f" (UID: "29d96f53-8b5e-4f32-adba-ac197a54931f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:22.137383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.137357    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d96f53-8b5e-4f32-adba-ac197a54931f-kube-api-access-v8bq9" pod "29d96f53-8b5e-4f32-adba-ac197a54931f" (UID: "29d96f53-8b5e-4f32-adba-ac197a54931f"). InnerVolumeSpecName "kube-api-access-v8bq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:22.146307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.146289    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:22.179307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.178972    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" path="/var/lib/kubelet/pods/117b4c77-4f1d-4b01-8665-f641b77b757c/volumes"
Oct 31 02:15:22.180498 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.179969    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a60988-774c-48a9-9a51-437c0e70c633" path="/var/lib/kubelet/pods/24a60988-774c-48a9-9a51-437c0e70c633/volumes"
Oct 31 02:15:22.181408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.180771    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" path="/var/lib/kubelet/pods/40530b75-ac11-4d5c-ad11-3e4f5726d972/volumes"
Oct 31 02:15:22.181789 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.181740    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" path="/var/lib/kubelet/pods/9b53e41f-48b3-4e5a-b600-8bca52c3526c/volumes"
Oct 31 02:15:22.182273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.182254    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" path="/var/lib/kubelet/pods/9e9990e4-da2e-4897-a3c1-4a9e9d7271b4/volumes"
Oct 31 02:15:22.182707 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.182683    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" path="/var/lib/kubelet/pods/a37eeab3-7312-4381-b0cb-55968b82c051/volumes"
Oct 31 02:15:22.184939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.183848    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" path="/var/lib/kubelet/pods/dc483cfa-c134-4176-b6b3-71af38093a2f/volumes"
Oct 31 02:15:22.236881 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.236845    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575a3759-7be6-416a-97bc-d2d5e6e96685-kube-api-access-p4k6j\" (UniqueName: \"kubernetes.io/projected/575a3759-7be6-416a-97bc-d2d5e6e96685-kube-api-access-p4k6j\") pod \"575a3759-7be6-416a-97bc-d2d5e6e96685\" (UID: \"575a3759-7be6-416a-97bc-d2d5e6e96685\") "
Oct 31 02:15:22.237386 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.237082    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575a3759-7be6-416a-97bc-d2d5e6e96685-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a3759-7be6-416a-97bc-d2d5e6e96685-maas-api-service-ca\") pod \"575a3759-7be6-416a-97bc-d2d5e6e96685\" (UID: \"575a3759-7be6-416a-97bc-d2d5e6e96685\") "
Oct 31 02:15:22.237458 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.237427    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575a3759-7be6-416a-97bc-d2d5e6e96685-maas-api-service-ca" pod "575a3759-7be6-416a-97bc-d2d5e6e96685" (UID: "575a3759-7be6-416a-97bc-d2d5e6e96685"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:22.237744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.237720    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8bq9\" (UniqueName: \"kubernetes.io/projected/29d96f53-8b5e-4f32-adba-ac197a54931f-kube-api-access-v8bq9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:22.237744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.237745    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d96f53-8b5e-4f32-adba-ac197a54931f-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:22.237932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.237759    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a3759-7be6-416a-97bc-d2d5e6e96685-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:22.239869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.239849    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575a3759-7be6-416a-97bc-d2d5e6e96685-kube-api-access-p4k6j" pod "575a3759-7be6-416a-97bc-d2d5e6e96685" (UID: "575a3759-7be6-416a-97bc-d2d5e6e96685"). InnerVolumeSpecName "kube-api-access-p4k6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:22.339780 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.339723    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4k6j\" (UniqueName: \"kubernetes.io/projected/575a3759-7be6-416a-97bc-d2d5e6e96685-kube-api-access-p4k6j\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:22.905506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.905477    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"
Oct 31 02:15:22.905828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.905504    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr" event={"ID":"575a3759-7be6-416a-97bc-d2d5e6e96685","Type":"ContainerDied","Data":"e4171dc52a09b4aa475b85f7a04bd02f8dcb8c1f2a656643bbae7b9ce186b6b4"}
Oct 31 02:15:22.905828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.905618    5293 scope.go:122] "RemoveContainer" containerID="b666eeb885efa2fa7be8114c1e2a345f658a5bf5ff650ffa432b81643abefa69"
Oct 31 02:15:22.913900 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.913872    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc" event={"ID":"29d96f53-8b5e-4f32-adba-ac197a54931f","Type":"ContainerDied","Data":"1d7803c8129017911e83ac837e90e5078517fe958dcc74b1d55486a9344fe147"}
Oct 31 02:15:22.914031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.913881    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"
Oct 31 02:15:22.919897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.919874    5293 scope.go:122] "RemoveContainer" containerID="417dc244270894668de54b5b1bd36cf5c741e5c9ea3736547efbaed4944b0610"
Oct 31 02:15:22.948218 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.948185    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"]
Oct 31 02:15:22.952179 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.952155    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bb5c51-29890215-q98wr"]
Oct 31 02:15:22.964871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.964700    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"]
Oct 31 02:15:22.968917 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:22.968888    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890215-ncqsc"]
Oct 31 02:15:23.085473 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.085445    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:23.158747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.158667    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8e42c7-d074-49ed-9355-b9cd77d0aace-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e42c7-d074-49ed-9355-b9cd77d0aace-maas-api-service-ca\") pod \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\" (UID: \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\") "
Oct 31 02:15:23.158925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.158847    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8e42c7-d074-49ed-9355-b9cd77d0aace-kube-api-access-24hh9\" (UniqueName: \"kubernetes.io/projected/7e8e42c7-d074-49ed-9355-b9cd77d0aace-kube-api-access-24hh9\") pod \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\" (UID: \"7e8e42c7-d074-49ed-9355-b9cd77d0aace\") "
Oct 31 02:15:23.159136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.159108    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8e42c7-d074-49ed-9355-b9cd77d0aace-maas-api-service-ca" pod "7e8e42c7-d074-49ed-9355-b9cd77d0aace" (UID: "7e8e42c7-d074-49ed-9355-b9cd77d0aace"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:23.159487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.159467    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e42c7-d074-49ed-9355-b9cd77d0aace-maas-api-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:23.161135 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.161111    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8e42c7-d074-49ed-9355-b9cd77d0aace-kube-api-access-24hh9" pod "7e8e42c7-d074-49ed-9355-b9cd77d0aace" (UID: "7e8e42c7-d074-49ed-9355-b9cd77d0aace"). InnerVolumeSpecName "kube-api-access-24hh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:23.293074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.267541    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24hh9\" (UniqueName: \"kubernetes.io/projected/7e8e42c7-d074-49ed-9355-b9cd77d0aace-kube-api-access-24hh9\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:23.926983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.926945    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"
Oct 31 02:15:23.926983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.926966    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j" event={"ID":"7e8e42c7-d074-49ed-9355-b9cd77d0aace","Type":"ContainerDied","Data":"1faa7d61b91f1e996ea68d9185ec98e75602f2f8756d11d32a43fc733bb4b06c"}
Oct 31 02:15:23.927253 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.927023    5293 scope.go:122] "RemoveContainer" containerID="f3f9e81097b1e92dae90d93f7d90d5a88e52487227a2fba7ddb4b2631ea1ff6c"
Oct 31 02:15:23.994132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.991194    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"]
Oct 31 02:15:23.999064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:23.998009    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890215-dcp8j"]
Oct 31 02:15:24.175406 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:24.175371    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" path="/var/lib/kubelet/pods/29d96f53-8b5e-4f32-adba-ac197a54931f/volumes"
Oct 31 02:15:24.175900 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:24.175865    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" path="/var/lib/kubelet/pods/575a3759-7be6-416a-97bc-d2d5e6e96685/volumes"
Oct 31 02:15:24.176433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:24.176416    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" path="/var/lib/kubelet/pods/7e8e42c7-d074-49ed-9355-b9cd77d0aace/volumes"
Oct 31 02:15:25.564324 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:25.562093    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"]
Oct 31 02:15:25.588679 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:25.588642    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6d7f5d-cf8cm"]
Oct 31 02:15:26.178138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:26.178105    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f485c7f-7eb0-44ba-976f-13d6b19d9101" path="/var/lib/kubelet/pods/3f485c7f-7eb0-44ba-976f-13d6b19d9101/volumes"
Oct 31 02:15:33.607066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:33.601387    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"]
Oct 31 02:15:33.607066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:33.601598    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" containerID="cri-o://bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a" gracePeriod=30
Oct 31 02:15:34.154433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.154402    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"]
Oct 31 02:15:34.155076 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155059    5293 cpu_manager.go:414] "containerMap: removing container" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:34.155136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155079    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:34.155136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155093    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy"
Oct 31 02:15:34.155136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155100    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy"
Oct 31 02:15:34.155136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155117    5293 cpu_manager.go:414] "containerMap: removing container" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:34.155136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155123    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:34.155136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155130    5293 cpu_manager.go:414] "containerMap: removing container" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:34.155136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155134    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155143    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3f485c7f-7eb0-44ba-976f-13d6b19d9101" containerName="revoke-keys"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155147    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f485c7f-7eb0-44ba-976f-13d6b19d9101" containerName="revoke-keys"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155156    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155161    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155168    5293 cpu_manager.go:414] "containerMap: removing container" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155173    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155178    5293 cpu_manager.go:414] "containerMap: removing container" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155182    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155192    5293 cpu_manager.go:414] "containerMap: removing container" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155196    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155208    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155212    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155218    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155222    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155230    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155234    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155241    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155246    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155251    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155255    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155264    5293 cpu_manager.go:414] "containerMap: removing container" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155268    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155276    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155280    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155286    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155290    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155299    5293 cpu_manager.go:414] "containerMap: removing container" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155303    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155311    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155315    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155326    5293 cpu_manager.go:414] "containerMap: removing container" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155330    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155338    5293 cpu_manager.go:414] "containerMap: removing container" podUID="58d4d72e-1061-4b38-bc36-ec9727d25e00" containerName="payload-processing"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155342    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d4d72e-1061-4b38-bc36-ec9727d25e00" containerName="payload-processing"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155349    5293 cpu_manager.go:414] "containerMap: removing container" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155354    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155360    5293 cpu_manager.go:414] "containerMap: removing container" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155364    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155371    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.155362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155375    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155385    5293 cpu_manager.go:414] "containerMap: removing container" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155389    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155408    5293 cpu_manager.go:414] "containerMap: removing container" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155413    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155421    5293 cpu_manager.go:414] "containerMap: removing container" podUID="43e2b864-2259-4446-89ad-0052cc9519b7" containerName="payload-processing"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155426    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e2b864-2259-4446-89ad-0052cc9519b7" containerName="payload-processing"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155527    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155534    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155542    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155549    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155554    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e2b864-2259-4446-89ad-0052cc9519b7" containerName="payload-processing"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155560    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155565    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155570    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155575    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155579    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f485c7f-7eb0-44ba-976f-13d6b19d9101" containerName="revoke-keys"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155584    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1321636-259d-41a9-9a08-99daca44e499" containerName="istio-proxy"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155590    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155594    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155600    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155605    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155609    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155614    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155619    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155624    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155629    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155634    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155639    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155643    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155648    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155653    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155658    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d4d72e-1061-4b38-bc36-ec9727d25e00" containerName="payload-processing"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155664    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d96f53-8b5e-4f32-adba-ac197a54931f" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155672    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37eeab3-7312-4381-b0cb-55968b82c051" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155677    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155683    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:34.156624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.155689    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:34.161114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.161093    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.185093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.183856    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"]
Oct 31 02:15:34.228063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.227705    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmk6k\" (UniqueName: \"kubernetes.io/projected/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-kube-api-access-rmk6k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.228063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.227795    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.228063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.227935    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.231837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.231795    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:15:34.232017 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.231899    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:15:34.339762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.336979    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.339762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.337088    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.339762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.337337    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmk6k\" (UniqueName: \"kubernetes.io/projected/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-kube-api-access-rmk6k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.351660 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.341972    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.352511 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.352479    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.384918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.384881    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmk6k\" (UniqueName: \"kubernetes.io/projected/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-kube-api-access-rmk6k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.480031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.479960    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:34.775315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:34.775291    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"]
Oct 31 02:15:34.776425 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:34.776402    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec665c14_3e52_4dc9_8fac_9be4b37a13c1.slice/crio-4e1cb15fd6fe6744593a6ece52c3a06d6f25931bb4993bd94972eb9d4ba8a8a1 WatchSource:0}: Error finding container 4e1cb15fd6fe6744593a6ece52c3a06d6f25931bb4993bd94972eb9d4ba8a8a1: Status 404 returned error can't find the container with id 4e1cb15fd6fe6744593a6ece52c3a06d6f25931bb4993bd94972eb9d4ba8a8a1
Oct 31 02:15:35.089555 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:35.089375    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k" event={"ID":"ec665c14-3e52-4dc9-8fac-9be4b37a13c1","Type":"ContainerStarted","Data":"4e1cb15fd6fe6744593a6ece52c3a06d6f25931bb4993bd94972eb9d4ba8a8a1"}
Oct 31 02:15:35.836556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:35.836502    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:15:35.836948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:35.836564    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:15:36.099159 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:36.099032    5293 generic.go:356] "Generic (PLEG): container finished" podID="ec665c14-3e52-4dc9-8fac-9be4b37a13c1" containerID="0c718d4305907ce2b1b0a2ea3f31a01cc254a993bdb17561dab85d180f50259f" exitCode=0
Oct 31 02:15:36.099159 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:36.099075    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k" event={"ID":"ec665c14-3e52-4dc9-8fac-9be4b37a13c1","Type":"ContainerDied","Data":"0c718d4305907ce2b1b0a2ea3f31a01cc254a993bdb17561dab85d180f50259f"}
Oct 31 02:15:37.322579 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.322547    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:37.400736 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.400706    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-xks-ca\") pod \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") "
Oct 31 02:15:37.400939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.400780    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-kube-api-access-rmk6k\" (UniqueName: \"kubernetes.io/projected/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-kube-api-access-rmk6k\") pod \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") "
Oct 31 02:15:37.400939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.400929    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-openshift-ca\") pod \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\" (UID: \"ec665c14-3e52-4dc9-8fac-9be4b37a13c1\") "
Oct 31 02:15:37.401897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.401848    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-openshift-ca" pod "ec665c14-3e52-4dc9-8fac-9be4b37a13c1" (UID: "ec665c14-3e52-4dc9-8fac-9be4b37a13c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:37.403698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.403672    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-xks-ca" pod "ec665c14-3e52-4dc9-8fac-9be4b37a13c1" (UID: "ec665c14-3e52-4dc9-8fac-9be4b37a13c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:15:37.404015 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.403992    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-kube-api-access-rmk6k" pod "ec665c14-3e52-4dc9-8fac-9be4b37a13c1" (UID: "ec665c14-3e52-4dc9-8fac-9be4b37a13c1"). InnerVolumeSpecName "kube-api-access-rmk6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:37.503631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.503600    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:37.503631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.503628    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmk6k\" (UniqueName: \"kubernetes.io/projected/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-kube-api-access-rmk6k\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:37.503631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:37.503638    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec665c14-3e52-4dc9-8fac-9be4b37a13c1-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:38.137482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.137445    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k" event={"ID":"ec665c14-3e52-4dc9-8fac-9be4b37a13c1","Type":"ContainerDied","Data":"4e1cb15fd6fe6744593a6ece52c3a06d6f25931bb4993bd94972eb9d4ba8a8a1"}
Oct 31 02:15:38.137698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.137500    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1cb15fd6fe6744593a6ece52c3a06d6f25931bb4993bd94972eb9d4ba8a8a1"
Oct 31 02:15:38.137698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.137553    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"
Oct 31 02:15:38.895809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.895714    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 02:15:38.931735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.931695    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-certs\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.932558 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.932496    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-socket\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.976070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.932278    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-certs" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:38.976350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.953210    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-socket" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:38.976531 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.976505    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-credential-socket\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.977234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.977108    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-credential-socket" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:38.977573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.977533    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-envoy\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.978345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.978294    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-data" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:38.978590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.978574    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-data\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.979210 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.979191    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-kube-api-access-fd6p7\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-kube-api-access-fd6p7\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.986437 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.986369    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201f0b07-540b-453c-a8c2-990aca029322-istiod-ca-cert" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:38.988474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.979476    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201f0b07-540b-453c-a8c2-990aca029322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201f0b07-540b-453c-a8c2-990aca029322-istiod-ca-cert\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.988474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.987030    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-istio-token\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-istio-token\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.988474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.987161    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/201f0b07-540b-453c-a8c2-990aca029322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201f0b07-540b-453c-a8c2-990aca029322-istio-podinfo\") pod \"201f0b07-540b-453c-a8c2-990aca029322\" (UID: \"201f0b07-540b-453c-a8c2-990aca029322\") "
Oct 31 02:15:38.988474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.988057    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:38.988474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.988080    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:38.988474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.988095    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:38.988474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.988108    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:38.988474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:38.988122    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201f0b07-540b-453c-a8c2-990aca029322-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:39.013365 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.011622    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-istio-token" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:39.035488 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.035422    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/201f0b07-540b-453c-a8c2-990aca029322-istio-podinfo" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:15:39.039384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.039327    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-envoy" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:39.076751 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.076693    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-kube-api-access-fd6p7" pod "201f0b07-540b-453c-a8c2-990aca029322" (UID: "201f0b07-540b-453c-a8c2-990aca029322"). InnerVolumeSpecName "kube-api-access-fd6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:39.103744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.096508    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201f0b07-540b-453c-a8c2-990aca029322-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:39.103744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.096551    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6p7\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-kube-api-access-fd6p7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:39.103744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.096568    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201f0b07-540b-453c-a8c2-990aca029322-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:39.103744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.096582    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201f0b07-540b-453c-a8c2-990aca029322-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:39.173449 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.173405    5293 generic.go:356] "Generic (PLEG): container finished" podID="201f0b07-540b-453c-a8c2-990aca029322" containerID="bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a" exitCode=0
Oct 31 02:15:39.173661 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.173631    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" event={"ID":"201f0b07-540b-453c-a8c2-990aca029322","Type":"ContainerDied","Data":"bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a"}
Oct 31 02:15:39.173722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.173678    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr" event={"ID":"201f0b07-540b-453c-a8c2-990aca029322","Type":"ContainerDied","Data":"3f38ffde6d3a992401dcb616e72927be22c92a8c9557f92727a7cc9b107fa4ed"}
Oct 31 02:15:39.173722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.173704    5293 scope.go:122] "RemoveContainer" containerID="bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a"
Oct 31 02:15:39.173886 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.173872    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"
Oct 31 02:15:39.254030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.253997    5293 scope.go:122] "RemoveContainer" containerID="4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3"
Oct 31 02:15:39.256077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.255406    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"]
Oct 31 02:15:39.261961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.261938    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6d7f5d-openshift-default-58d959544c-x4cvr"]
Oct 31 02:15:39.272969 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.272862    5293 scope.go:122] "RemoveContainer" containerID="bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a"
Oct 31 02:15:39.273235 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:39.273212    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a\": container with ID starting with bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a not found: ID does not exist" containerID="bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a"
Oct 31 02:15:39.273333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.273245    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a"} err="failed to get container status \"bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a\": rpc error: code = NotFound desc = could not find container \"bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a\": container with ID starting with bdee8517771837abea74d6895a319d633124d5eaeff99a4cf69665329fc12a7a not found: ID does not exist"
Oct 31 02:15:39.273333 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.273265    5293 scope.go:122] "RemoveContainer" containerID="4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3"
Oct 31 02:15:39.273600 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:39.273567    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3\": container with ID starting with 4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3 not found: ID does not exist" containerID="4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3"
Oct 31 02:15:39.273667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:39.273612    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3"} err="failed to get container status \"4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3\": rpc error: code = NotFound desc = could not find container \"4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3\": container with ID starting with 4fcdc393ade86662a8c4acefdf1ab1074213f082f84c03b76bdbdaaf8f3fafd3 not found: ID does not exist"
Oct 31 02:15:40.175033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:40.174998    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201f0b07-540b-453c-a8c2-990aca029322" path="/var/lib/kubelet/pods/201f0b07-540b-453c-a8c2-990aca029322/volumes"
Oct 31 02:15:41.779272 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:41.779236    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:15:44.343990 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:44.343956    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"]
Oct 31 02:15:48.045215 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:48.045124    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"]
Oct 31 02:15:48.064644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:48.064605    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e57add-b699k"]
Oct 31 02:15:48.174868 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:48.174832    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec665c14-3e52-4dc9-8fac-9be4b37a13c1" path="/var/lib/kubelet/pods/ec665c14-3e52-4dc9-8fac-9be4b37a13c1/volumes"
Oct 31 02:15:49.228160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.228125    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:15:49.344981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.344913    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq" podUID="fe286755-5d6e-43ec-b73f-dc46b359b9ab" containerName="payload-processing" containerID="cri-o://ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584" gracePeriod=25
Oct 31 02:15:49.625416 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.625389    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 02:15:49.675508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.674605    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe286755-5d6e-43ec-b73f-dc46b359b9ab-kube-api-access-wj5wb\" (UniqueName: \"kubernetes.io/projected/fe286755-5d6e-43ec-b73f-dc46b359b9ab-kube-api-access-wj5wb\") pod \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\" (UID: \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\") "
Oct 31 02:15:49.675508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.674757    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe286755-5d6e-43ec-b73f-dc46b359b9ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe286755-5d6e-43ec-b73f-dc46b359b9ab-plugins-config-volume\") pod \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\" (UID: \"fe286755-5d6e-43ec-b73f-dc46b359b9ab\") "
Oct 31 02:15:49.695724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.695667    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe286755-5d6e-43ec-b73f-dc46b359b9ab-kube-api-access-wj5wb" pod "fe286755-5d6e-43ec-b73f-dc46b359b9ab" (UID: "fe286755-5d6e-43ec-b73f-dc46b359b9ab"). InnerVolumeSpecName "kube-api-access-wj5wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:49.707383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.707315    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe286755-5d6e-43ec-b73f-dc46b359b9ab-plugins-config-volume" pod "fe286755-5d6e-43ec-b73f-dc46b359b9ab" (UID: "fe286755-5d6e-43ec-b73f-dc46b359b9ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:49.776721 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.776646    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5wb\" (UniqueName: \"kubernetes.io/projected/fe286755-5d6e-43ec-b73f-dc46b359b9ab-kube-api-access-wj5wb\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:49.776721 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:49.776676    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe286755-5d6e-43ec-b73f-dc46b359b9ab-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:50.291895 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.291864    5293 generic.go:356] "Generic (PLEG): container finished" podID="fe286755-5d6e-43ec-b73f-dc46b359b9ab" containerID="ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584" exitCode=0
Oct 31 02:15:50.292277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.292123    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq" event={"ID":"fe286755-5d6e-43ec-b73f-dc46b359b9ab","Type":"ContainerDied","Data":"ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584"}
Oct 31 02:15:50.292277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.292163    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq" event={"ID":"fe286755-5d6e-43ec-b73f-dc46b359b9ab","Type":"ContainerDied","Data":"c1b25b75291bc1b524b205c37060c27585bc6fc991aeec6a6c82958cea0af8fe"}
Oct 31 02:15:50.292277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.292182    5293 scope.go:122] "RemoveContainer" containerID="ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584"
Oct 31 02:15:50.292389 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.292304    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"
Oct 31 02:15:50.306275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.306247    5293 scope.go:122] "RemoveContainer" containerID="ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584"
Oct 31 02:15:50.306551 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:15:50.306529    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584\": container with ID starting with ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584 not found: ID does not exist" containerID="ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584"
Oct 31 02:15:50.306640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.306559    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584"} err="failed to get container status \"ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584\": rpc error: code = NotFound desc = could not find container \"ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584\": container with ID starting with ac7e38a0f721b08e209c598d2c6b9acae6b770abf824e1719f7c099fa77b3584 not found: ID does not exist"
Oct 31 02:15:50.313961 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.313937    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"]
Oct 31 02:15:50.320621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.320597    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e57add-7cb8c9f9bc-nvmsq"]
Oct 31 02:15:50.563987 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.563899    5293 scope.go:122] "RemoveContainer" containerID="d43c853cb2b8e9a0a466dc8eac3e70d880d69c7e187f7bf1df75177031a0e799"
Oct 31 02:15:50.575722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:50.575698    5293 scope.go:122] "RemoveContainer" containerID="c7142f62f24a30b9ed51ffba0a796d5372080c46bb6c61c0056814b21a545a13"
Oct 31 02:15:52.175338 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:52.175302    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe286755-5d6e-43ec-b73f-dc46b359b9ab" path="/var/lib/kubelet/pods/fe286755-5d6e-43ec-b73f-dc46b359b9ab/volumes"
Oct 31 02:15:56.035816 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.035781    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"]
Oct 31 02:15:56.036703 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036680    5293 cpu_manager.go:414] "containerMap: removing container" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:56.036814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036705    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:56.036814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036721    5293 cpu_manager.go:414] "containerMap: removing container" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy"
Oct 31 02:15:56.036814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036728    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy"
Oct 31 02:15:56.036814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036740    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ec665c14-3e52-4dc9-8fac-9be4b37a13c1" containerName="revoke-keys"
Oct 31 02:15:56.036814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036747    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec665c14-3e52-4dc9-8fac-9be4b37a13c1" containerName="revoke-keys"
Oct 31 02:15:56.036814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036764    5293 cpu_manager.go:414] "containerMap: removing container" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036821    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036849    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036856    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8e42c7-d074-49ed-9355-b9cd77d0aace" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036870    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036877    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9990e4-da2e-4897-a3c1-4a9e9d7271b4" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036893    5293 cpu_manager.go:414] "containerMap: removing container" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036902    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc483cfa-c134-4176-b6b3-71af38093a2f" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036912    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036920    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036935    5293 cpu_manager.go:414] "containerMap: removing container" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036943    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a60988-774c-48a9-9a51-437c0e70c633" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036955    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fe286755-5d6e-43ec-b73f-dc46b359b9ab" containerName="payload-processing"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036962    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe286755-5d6e-43ec-b73f-dc46b359b9ab" containerName="payload-processing"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.036993    5293 cpu_manager.go:414] "containerMap: removing container" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:56.037182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037000    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:56.037954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037215    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b53e41f-48b3-4e5a-b600-8bca52c3526c" containerName="cleanup"
Oct 31 02:15:56.037954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037232    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="40530b75-ac11-4d5c-ad11-3e4f5726d972" containerName="cleanup"
Oct 31 02:15:56.037954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037244    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="575a3759-7be6-416a-97bc-d2d5e6e96685" containerName="cleanup"
Oct 31 02:15:56.037954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037255    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy"
Oct 31 02:15:56.037954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037264    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec665c14-3e52-4dc9-8fac-9be4b37a13c1" containerName="revoke-keys"
Oct 31 02:15:56.037954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037275    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="117b4c77-4f1d-4b01-8665-f641b77b757c" containerName="cleanup"
Oct 31 02:15:56.037954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037287    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy"
Oct 31 02:15:56.037954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.037296    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe286755-5d6e-43ec-b73f-dc46b359b9ab" containerName="payload-processing"
Oct 31 02:15:56.050211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.050185    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.058273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.057394    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:15:56.058273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.058077    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:15:56.060433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.060408    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"]
Oct 31 02:15:56.188257 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.188223    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.188443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.188280    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.188443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.188362    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55b6\" (UniqueName: \"kubernetes.io/projected/95c92982-f897-41c1-b377-93c8cb7d55d1-kube-api-access-k55b6\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.290374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.290278    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.290565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.290385    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.290667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.290631    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55b6\" (UniqueName: \"kubernetes.io/projected/95c92982-f897-41c1-b377-93c8cb7d55d1-kube-api-access-k55b6\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.291286 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.291261    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.292909 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.292885    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.302300 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.302274    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55b6\" (UniqueName: \"kubernetes.io/projected/95c92982-f897-41c1-b377-93c8cb7d55d1-kube-api-access-k55b6\") pod \"maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.381081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.381031    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:56.671762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:56.671728    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"]
Oct 31 02:15:56.673231 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:15:56.673200    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c92982_f897_41c1_b377_93c8cb7d55d1.slice/crio-5184fcdc5ee4064b7a8a2785b88cf843466d4f399f53f48e05bcb9854c51f355 WatchSource:0}: Error finding container 5184fcdc5ee4064b7a8a2785b88cf843466d4f399f53f48e05bcb9854c51f355: Status 404 returned error can't find the container with id 5184fcdc5ee4064b7a8a2785b88cf843466d4f399f53f48e05bcb9854c51f355
Oct 31 02:15:57.352659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:57.352621    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx" event={"ID":"95c92982-f897-41c1-b377-93c8cb7d55d1","Type":"ContainerStarted","Data":"5614c7c27016395a486b8517a4efba2252d15e633eeca4c137beac54f3970b2b"}
Oct 31 02:15:57.353151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:57.352680    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx" event={"ID":"95c92982-f897-41c1-b377-93c8cb7d55d1","Type":"ContainerStarted","Data":"5184fcdc5ee4064b7a8a2785b88cf843466d4f399f53f48e05bcb9854c51f355"}
Oct 31 02:15:57.370682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:57.370580    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx" podStartSLOduration=1.370564324 podStartE2EDuration="1.370564324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:57.369377568 +0000 UTC m=+3315.318025961" watchObservedRunningTime="2026-10-31 02:15:57.370564324 +0000 UTC m=+3315.319212732"
Oct 31 02:15:58.361412 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:58.361378    5293 generic.go:356] "Generic (PLEG): container finished" podID="95c92982-f897-41c1-b377-93c8cb7d55d1" containerID="5614c7c27016395a486b8517a4efba2252d15e633eeca4c137beac54f3970b2b" exitCode=0
Oct 31 02:15:58.361902 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:58.361470    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx" event={"ID":"95c92982-f897-41c1-b377-93c8cb7d55d1","Type":"ContainerDied","Data":"5614c7c27016395a486b8517a4efba2252d15e633eeca4c137beac54f3970b2b"}
Oct 31 02:15:59.537007 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.536974    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:15:59.596844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.592763    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-xks-ca\") pod \"95c92982-f897-41c1-b377-93c8cb7d55d1\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") "
Oct 31 02:15:59.596844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.592853    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c92982-f897-41c1-b377-93c8cb7d55d1-kube-api-access-k55b6\" (UniqueName: \"kubernetes.io/projected/95c92982-f897-41c1-b377-93c8cb7d55d1-kube-api-access-k55b6\") pod \"95c92982-f897-41c1-b377-93c8cb7d55d1\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") "
Oct 31 02:15:59.596844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.593268    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-openshift-ca\") pod \"95c92982-f897-41c1-b377-93c8cb7d55d1\" (UID: \"95c92982-f897-41c1-b377-93c8cb7d55d1\") "
Oct 31 02:15:59.638064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.626240    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-openshift-ca" pod "95c92982-f897-41c1-b377-93c8cb7d55d1" (UID: "95c92982-f897-41c1-b377-93c8cb7d55d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:59.642428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.638775    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-xks-ca" pod "95c92982-f897-41c1-b377-93c8cb7d55d1" (UID: "95c92982-f897-41c1-b377-93c8cb7d55d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:15:59.658400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.655945    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c92982-f897-41c1-b377-93c8cb7d55d1-kube-api-access-k55b6" pod "95c92982-f897-41c1-b377-93c8cb7d55d1" (UID: "95c92982-f897-41c1-b377-93c8cb7d55d1"). InnerVolumeSpecName "kube-api-access-k55b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:59.695272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.695225    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:59.695272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.695276    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k55b6\" (UniqueName: \"kubernetes.io/projected/95c92982-f897-41c1-b377-93c8cb7d55d1-kube-api-access-k55b6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:59.695468 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:15:59.695292    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c92982-f897-41c1-b377-93c8cb7d55d1-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.378393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:00.378366    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"
Oct 31 02:16:00.378596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:00.378365    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx" event={"ID":"95c92982-f897-41c1-b377-93c8cb7d55d1","Type":"ContainerDied","Data":"5184fcdc5ee4064b7a8a2785b88cf843466d4f399f53f48e05bcb9854c51f355"}
Oct 31 02:16:00.378596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:00.378513    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5184fcdc5ee4064b7a8a2785b88cf843466d4f399f53f48e05bcb9854c51f355"
Oct 31 02:16:00.802074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:00.802004    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"]
Oct 31 02:16:06.306983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.306942    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"]
Oct 31 02:16:06.316685 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.316658    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"]
Oct 31 02:16:06.509444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.509408    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"]
Oct 31 02:16:06.510156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.510139    5293 cpu_manager.go:414] "containerMap: removing container" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy"
Oct 31 02:16:06.510208 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.510158    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201f0b07-540b-453c-a8c2-990aca029322" containerName="istio-proxy"
Oct 31 02:16:06.510208 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.510183    5293 cpu_manager.go:414] "containerMap: removing container" podUID="95c92982-f897-41c1-b377-93c8cb7d55d1" containerName="revoke-keys"
Oct 31 02:16:06.510208 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.510188    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c92982-f897-41c1-b377-93c8cb7d55d1" containerName="revoke-keys"
Oct 31 02:16:06.510317 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.510285    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c92982-f897-41c1-b377-93c8cb7d55d1" containerName="revoke-keys"
Oct 31 02:16:06.514395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.514370    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.518484 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.518456    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:16:06.520548 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.520521    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:16:06.532793 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.532751    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"]
Oct 31 02:16:06.537994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.537964    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.538169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.538089    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4r82\" (UniqueName: \"kubernetes.io/projected/7ef03134-9a32-466a-a379-e533ba5e8a5f-kube-api-access-w4r82\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.538275 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.538245    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.700063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.680542    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.700063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.680660    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4r82\" (UniqueName: \"kubernetes.io/projected/7ef03134-9a32-466a-a379-e533ba5e8a5f-kube-api-access-w4r82\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.700063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.680821    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.701012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.700946    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.742063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.733100    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.811889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.811809    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4r82\" (UniqueName: \"kubernetes.io/projected/7ef03134-9a32-466a-a379-e533ba5e8a5f-kube-api-access-w4r82\") pod \"maas-api-revoke-sub-0f4ecdd9-8xw62\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:06.860571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:06.860539    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:07.234605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:07.234524    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"]
Oct 31 02:16:07.241121 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:16:07.241025    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef03134_9a32_466a_a379_e533ba5e8a5f.slice/crio-dd98c4151d3bfe00e940b7d0e63e5c8832aa2b83e1fe2627e2895d50e3276a4f WatchSource:0}: Error finding container dd98c4151d3bfe00e940b7d0e63e5c8832aa2b83e1fe2627e2895d50e3276a4f: Status 404 returned error can't find the container with id dd98c4151d3bfe00e940b7d0e63e5c8832aa2b83e1fe2627e2895d50e3276a4f
Oct 31 02:16:07.458495 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:07.458444    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62" event={"ID":"7ef03134-9a32-466a-a379-e533ba5e8a5f","Type":"ContainerStarted","Data":"dd98c4151d3bfe00e940b7d0e63e5c8832aa2b83e1fe2627e2895d50e3276a4f"}
Oct 31 02:16:07.959168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:07.959137    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"]
Oct 31 02:16:07.963891 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:07.963870    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:07.974742 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:07.974714    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"]
Oct 31 02:16:08.055724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.055669    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.055929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.055787    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.056011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.055950    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjh8p\" (UniqueName: \"kubernetes.io/projected/415bd3ec-d41e-4ee9-94c2-340f52c1b406-kube-api-access-mjh8p\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.157817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.157778    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjh8p\" (UniqueName: \"kubernetes.io/projected/415bd3ec-d41e-4ee9-94c2-340f52c1b406-kube-api-access-mjh8p\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.157978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.157949    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.158073 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.157992    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.158624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.158605    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.160506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.160487    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.169582 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.169558    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjh8p\" (UniqueName: \"kubernetes.io/projected/415bd3ec-d41e-4ee9-94c2-340f52c1b406-kube-api-access-mjh8p\") pod \"maas-api-revoke-sub-a19e4613-qkmrh\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.282616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.282589    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:08.479091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.478970    5293 generic.go:356] "Generic (PLEG): container finished" podID="7ef03134-9a32-466a-a379-e533ba5e8a5f" containerID="7f2efe141750a86630d3f4dd5cffc90baecb96ffc082d45300abf0e24c50b025" exitCode=0
Oct 31 02:16:08.479091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.479020    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62" event={"ID":"7ef03134-9a32-466a-a379-e533ba5e8a5f","Type":"ContainerDied","Data":"7f2efe141750a86630d3f4dd5cffc90baecb96ffc082d45300abf0e24c50b025"}
Oct 31 02:16:08.573708 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:08.573681    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"]
Oct 31 02:16:08.573816 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:16:08.573719    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415bd3ec_d41e_4ee9_94c2_340f52c1b406.slice/crio-acfd34f9ee6b9248b7a36a77d03e8fa7a44f21526b8eec6ed7cb068ac06c1593 WatchSource:0}: Error finding container acfd34f9ee6b9248b7a36a77d03e8fa7a44f21526b8eec6ed7cb068ac06c1593: Status 404 returned error can't find the container with id acfd34f9ee6b9248b7a36a77d03e8fa7a44f21526b8eec6ed7cb068ac06c1593
Oct 31 02:16:09.489195 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.489159    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh" event={"ID":"415bd3ec-d41e-4ee9-94c2-340f52c1b406","Type":"ContainerStarted","Data":"00a9c69419b53fc5caebc17595654aeb7ca3e50c6daf694e93c9c6fb95e2fd27"}
Oct 31 02:16:09.489594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.489222    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh" event={"ID":"415bd3ec-d41e-4ee9-94c2-340f52c1b406","Type":"ContainerStarted","Data":"acfd34f9ee6b9248b7a36a77d03e8fa7a44f21526b8eec6ed7cb068ac06c1593"}
Oct 31 02:16:09.511356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.511288    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh" podStartSLOduration=2.511264912 podStartE2EDuration="2.511264912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:09.507328785 +0000 UTC m=+3327.455977177" watchObservedRunningTime="2026-10-31 02:16:09.511264912 +0000 UTC m=+3327.459913307"
Oct 31 02:16:09.696827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.696800    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:09.792675 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.792591    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-xks-ca\") pod \"7ef03134-9a32-466a-a379-e533ba5e8a5f\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") "
Oct 31 02:16:09.792827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.792703    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-openshift-ca\") pod \"7ef03134-9a32-466a-a379-e533ba5e8a5f\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") "
Oct 31 02:16:09.792827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.792745    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef03134-9a32-466a-a379-e533ba5e8a5f-kube-api-access-w4r82\" (UniqueName: \"kubernetes.io/projected/7ef03134-9a32-466a-a379-e533ba5e8a5f-kube-api-access-w4r82\") pod \"7ef03134-9a32-466a-a379-e533ba5e8a5f\" (UID: \"7ef03134-9a32-466a-a379-e533ba5e8a5f\") "
Oct 31 02:16:09.793160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.793130    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-openshift-ca" pod "7ef03134-9a32-466a-a379-e533ba5e8a5f" (UID: "7ef03134-9a32-466a-a379-e533ba5e8a5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:09.793456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.793439    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:09.794802 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.794784    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-xks-ca" pod "7ef03134-9a32-466a-a379-e533ba5e8a5f" (UID: "7ef03134-9a32-466a-a379-e533ba5e8a5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:09.794916 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.794906    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef03134-9a32-466a-a379-e533ba5e8a5f-kube-api-access-w4r82" pod "7ef03134-9a32-466a-a379-e533ba5e8a5f" (UID: "7ef03134-9a32-466a-a379-e533ba5e8a5f"). InnerVolumeSpecName "kube-api-access-w4r82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:09.894785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.894750    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4r82\" (UniqueName: \"kubernetes.io/projected/7ef03134-9a32-466a-a379-e533ba5e8a5f-kube-api-access-w4r82\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:09.894785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:09.894780    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef03134-9a32-466a-a379-e533ba5e8a5f-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:10.499675 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:10.499577    5293 generic.go:356] "Generic (PLEG): container finished" podID="415bd3ec-d41e-4ee9-94c2-340f52c1b406" containerID="00a9c69419b53fc5caebc17595654aeb7ca3e50c6daf694e93c9c6fb95e2fd27" exitCode=0
Oct 31 02:16:10.499675 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:10.499611    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh" event={"ID":"415bd3ec-d41e-4ee9-94c2-340f52c1b406","Type":"ContainerDied","Data":"00a9c69419b53fc5caebc17595654aeb7ca3e50c6daf694e93c9c6fb95e2fd27"}
Oct 31 02:16:10.501450 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:10.501424    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62"
Oct 31 02:16:10.501596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:10.501444    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4ecdd9-8xw62" event={"ID":"7ef03134-9a32-466a-a379-e533ba5e8a5f","Type":"ContainerDied","Data":"dd98c4151d3bfe00e940b7d0e63e5c8832aa2b83e1fe2627e2895d50e3276a4f"}
Oct 31 02:16:10.501596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:10.501488    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd98c4151d3bfe00e940b7d0e63e5c8832aa2b83e1fe2627e2895d50e3276a4f"
Oct 31 02:16:11.308194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.308153    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929" podUID="daae2c1d-05e9-4a5f-892a-2f454f4b1f76" containerName="payload-processing" containerID="cri-o://ca0b1f4b59afd4f8563909be2d63776daa6c07f674ca54e579dbfc35cbef1c78" gracePeriod=25
Oct 31 02:16:11.317618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.317589    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6" podUID="be9d1606-5c4a-4d16-8370-958320546e86" containerName="payload-pre-processing" containerID="cri-o://a95a3da163e34fd9c9813dad9d604445d588a90d9532d56358a0286023b07be5" gracePeriod=25
Oct 31 02:16:11.687601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.687576    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:11.779977 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:16:11.779936    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:16:11.826847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.826721    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415bd3ec-d41e-4ee9-94c2-340f52c1b406-kube-api-access-mjh8p\" (UniqueName: \"kubernetes.io/projected/415bd3ec-d41e-4ee9-94c2-340f52c1b406-kube-api-access-mjh8p\") pod \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") "
Oct 31 02:16:11.827031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.827003    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-xks-ca\") pod \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") "
Oct 31 02:16:11.827134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.827078    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-openshift-ca\") pod \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\" (UID: \"415bd3ec-d41e-4ee9-94c2-340f52c1b406\") "
Oct 31 02:16:11.827605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.827532    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-openshift-ca" pod "415bd3ec-d41e-4ee9-94c2-340f52c1b406" (UID: "415bd3ec-d41e-4ee9-94c2-340f52c1b406"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:11.827839 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.827818    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:11.829265 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.829236    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415bd3ec-d41e-4ee9-94c2-340f52c1b406-kube-api-access-mjh8p" pod "415bd3ec-d41e-4ee9-94c2-340f52c1b406" (UID: "415bd3ec-d41e-4ee9-94c2-340f52c1b406"). InnerVolumeSpecName "kube-api-access-mjh8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:11.829645 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.829622    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-xks-ca" pod "415bd3ec-d41e-4ee9-94c2-340f52c1b406" (UID: "415bd3ec-d41e-4ee9-94c2-340f52c1b406"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:11.928858 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.928827    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjh8p\" (UniqueName: \"kubernetes.io/projected/415bd3ec-d41e-4ee9-94c2-340f52c1b406-kube-api-access-mjh8p\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:11.928858 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:11.928858    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415bd3ec-d41e-4ee9-94c2-340f52c1b406-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:12.522611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:12.522526    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh"
Oct 31 02:16:12.522611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:12.522551    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19e4613-qkmrh" event={"ID":"415bd3ec-d41e-4ee9-94c2-340f52c1b406","Type":"ContainerDied","Data":"acfd34f9ee6b9248b7a36a77d03e8fa7a44f21526b8eec6ed7cb068ac06c1593"}
Oct 31 02:16:12.522611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:12.522609    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acfd34f9ee6b9248b7a36a77d03e8fa7a44f21526b8eec6ed7cb068ac06c1593"
Oct 31 02:16:13.148905 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:13.148861    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6" podUID="be9d1606-5c4a-4d16-8370-958320546e86" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:16:13.150206 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:13.149023    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929" podUID="daae2c1d-05e9-4a5f-892a-2f454f4b1f76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:16:15.824841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.824787    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main" containerID="cri-o://834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132" gracePeriod=45
Oct 31 02:16:15.834993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.834959    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"]
Oct 31 02:16:15.835719 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.835702    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7ef03134-9a32-466a-a379-e533ba5e8a5f" containerName="revoke-keys"
Oct 31 02:16:15.835811 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.835720    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef03134-9a32-466a-a379-e533ba5e8a5f" containerName="revoke-keys"
Oct 31 02:16:15.835811 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.835736    5293 cpu_manager.go:414] "containerMap: removing container" podUID="415bd3ec-d41e-4ee9-94c2-340f52c1b406" containerName="revoke-keys"
Oct 31 02:16:15.835811 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.835743    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415bd3ec-d41e-4ee9-94c2-340f52c1b406" containerName="revoke-keys"
Oct 31 02:16:15.835971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.835831    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef03134-9a32-466a-a379-e533ba5e8a5f" containerName="revoke-keys"
Oct 31 02:16:15.835971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.835843    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="415bd3ec-d41e-4ee9-94c2-340f52c1b406" containerName="revoke-keys"
Oct 31 02:16:15.839491 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.839469    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:15.846608 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.846579    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"]
Oct 31 02:16:15.848460 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.848435    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:16:15.849156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.848649    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:16:15.852458 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.852436    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:15.852887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.852862    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"]
Oct 31 02:16:15.859476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:15.859454    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"]
Oct 31 02:16:16.003240 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.003208    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.003393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.003273    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.003501 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.003470    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.003613 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.003590    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmz5g\" (UniqueName: \"kubernetes.io/projected/26eb0659-7c39-451d-8e0b-7c58b85908a8-kube-api-access-nmz5g\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.003758 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.003737    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.003888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.003867    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs282\" (UniqueName: \"kubernetes.io/projected/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-kube-api-access-fs282\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.104998 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.104964    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.105166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.105053    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.105166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.105137    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.105286 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.105206    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmz5g\" (UniqueName: \"kubernetes.io/projected/26eb0659-7c39-451d-8e0b-7c58b85908a8-kube-api-access-nmz5g\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.105347 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.105330    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.105772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.105426    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs282\" (UniqueName: \"kubernetes.io/projected/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-kube-api-access-fs282\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.105889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.105811    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.106252 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.106202    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.107946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.107925    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.108106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.108088    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.117422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.117395    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmz5g\" (UniqueName: \"kubernetes.io/projected/26eb0659-7c39-451d-8e0b-7c58b85908a8-kube-api-access-nmz5g\") pod \"maas-api-revoke-sub-57403e6a-5pfsz\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.118314 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.118293    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs282\" (UniqueName: \"kubernetes.io/projected/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-kube-api-access-fs282\") pod \"maas-api-revoke-sub-17e82121-kwc96\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.150532 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.150496    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:16:16.271065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.247330    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:16.272848 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.254268    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:16.372442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.372185    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-tmp-dir\") pod \"c7e17ef2-7de5-45d9-85cd-a339307d8477\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") "
Oct 31 02:16:16.372442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.372239    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e17ef2-7de5-45d9-85cd-a339307d8477-kube-api-access-486bj\" (UniqueName: \"kubernetes.io/projected/c7e17ef2-7de5-45d9-85cd-a339307d8477-kube-api-access-486bj\") pod \"c7e17ef2-7de5-45d9-85cd-a339307d8477\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") "
Oct 31 02:16:16.372442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.372295    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7e17ef2-7de5-45d9-85cd-a339307d8477-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7e17ef2-7de5-45d9-85cd-a339307d8477-tls-certs\") pod \"c7e17ef2-7de5-45d9-85cd-a339307d8477\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") "
Oct 31 02:16:16.372442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.372375    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-home\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-home\") pod \"c7e17ef2-7de5-45d9-85cd-a339307d8477\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") "
Oct 31 02:16:16.372780 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.372665    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-model-cache\") pod \"c7e17ef2-7de5-45d9-85cd-a339307d8477\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") "
Oct 31 02:16:16.372780 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.372775    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-dshm\") pod \"c7e17ef2-7de5-45d9-85cd-a339307d8477\" (UID: \"c7e17ef2-7de5-45d9-85cd-a339307d8477\") "
Oct 31 02:16:16.380100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.377202    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-tmp-dir" pod "c7e17ef2-7de5-45d9-85cd-a339307d8477" (UID: "c7e17ef2-7de5-45d9-85cd-a339307d8477"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.380100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.377238    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-model-cache" pod "c7e17ef2-7de5-45d9-85cd-a339307d8477" (UID: "c7e17ef2-7de5-45d9-85cd-a339307d8477"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.380100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.377688    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-home" pod "c7e17ef2-7de5-45d9-85cd-a339307d8477" (UID: "c7e17ef2-7de5-45d9-85cd-a339307d8477"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.380100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.377935    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e17ef2-7de5-45d9-85cd-a339307d8477-kube-api-access-486bj" pod "c7e17ef2-7de5-45d9-85cd-a339307d8477" (UID: "c7e17ef2-7de5-45d9-85cd-a339307d8477"). InnerVolumeSpecName "kube-api-access-486bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:16.380420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.380341    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-dshm" pod "c7e17ef2-7de5-45d9-85cd-a339307d8477" (UID: "c7e17ef2-7de5-45d9-85cd-a339307d8477"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.390863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.380759    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e17ef2-7de5-45d9-85cd-a339307d8477-tls-certs" pod "c7e17ef2-7de5-45d9-85cd-a339307d8477" (UID: "c7e17ef2-7de5-45d9-85cd-a339307d8477"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:16.475804 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.475769    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.475804 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.475807    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.476084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.475823    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.476084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.475837    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-486bj\" (UniqueName: \"kubernetes.io/projected/c7e17ef2-7de5-45d9-85cd-a339307d8477-kube-api-access-486bj\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.476084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.475851    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7e17ef2-7de5-45d9-85cd-a339307d8477-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.476084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.475864    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7e17ef2-7de5-45d9-85cd-a339307d8477-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.559303 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.559267    5293 generic.go:356] "Generic (PLEG): container finished" podID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerID="834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132" exitCode=0
Oct 31 02:16:16.559470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.559329    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"
Oct 31 02:16:16.559470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.559355    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" event={"ID":"c7e17ef2-7de5-45d9-85cd-a339307d8477","Type":"ContainerDied","Data":"834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132"}
Oct 31 02:16:16.559470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.559403    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq" event={"ID":"c7e17ef2-7de5-45d9-85cd-a339307d8477","Type":"ContainerDied","Data":"46db11f7c24d26d1be207bc79b11708ca2a3e62983c230f561e883ff89965d3e"}
Oct 31 02:16:16.559470 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.559423    5293 scope.go:122] "RemoveContainer" containerID="834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132"
Oct 31 02:16:16.571462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.571445    5293 scope.go:122] "RemoveContainer" containerID="834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132"
Oct 31 02:16:16.571763 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:16:16.571741    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132\": container with ID starting with 834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132 not found: ID does not exist" containerID="834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132"
Oct 31 02:16:16.571846 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.571777    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132"} err="failed to get container status \"834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132\": rpc error: code = NotFound desc = could not find container \"834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132\": container with ID starting with 834f8c0f228056bd9ffe2a97a2d566a0be5dbfb5a6ef424464b1d4b6a411f132 not found: ID does not exist"
Oct 31 02:16:16.591203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.591172    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"]
Oct 31 02:16:16.594725 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.594703    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/premium-w0-3cf6f4-kserve-5dcd9cbcf4-5zvgq"]
Oct 31 02:16:16.632750 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.632721    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"]
Oct 31 02:16:16.633693 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:16:16.633669    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26eb0659_7c39_451d_8e0b_7c58b85908a8.slice/crio-8ef4a4d2bec8c8cb7ec9b79c4cb38195f8ab8dfbb29bff47a41a8cfc47229f80 WatchSource:0}: Error finding container 8ef4a4d2bec8c8cb7ec9b79c4cb38195f8ab8dfbb29bff47a41a8cfc47229f80: Status 404 returned error can't find the container with id 8ef4a4d2bec8c8cb7ec9b79c4cb38195f8ab8dfbb29bff47a41a8cfc47229f80
Oct 31 02:16:16.658554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:16.658526    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"]
Oct 31 02:16:16.677422 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:16:16.677397    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6538fad_9fb0_42e3_9e97_d54de4ebafc6.slice/crio-4a4ae3c1eb27a073b8c430b9b0c2d77d545694df4f45c2d2b807868a64f35640 WatchSource:0}: Error finding container 4a4ae3c1eb27a073b8c430b9b0c2d77d545694df4f45c2d2b807868a64f35640: Status 404 returned error can't find the container with id 4a4ae3c1eb27a073b8c430b9b0c2d77d545694df4f45c2d2b807868a64f35640
Oct 31 02:16:17.570762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:17.570713    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz" event={"ID":"26eb0659-7c39-451d-8e0b-7c58b85908a8","Type":"ContainerStarted","Data":"a3b550cf42f6946b26a41753b6dcbd0b7e06385f98341174ade72b7e55c4a961"}
Oct 31 02:16:17.571348 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:17.570803    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz" event={"ID":"26eb0659-7c39-451d-8e0b-7c58b85908a8","Type":"ContainerStarted","Data":"8ef4a4d2bec8c8cb7ec9b79c4cb38195f8ab8dfbb29bff47a41a8cfc47229f80"}
Oct 31 02:16:17.572855 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:17.572825    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96" event={"ID":"e6538fad-9fb0-42e3-9e97-d54de4ebafc6","Type":"ContainerStarted","Data":"5d0bd0d9d05c508bc25683fb0eefc10eafae5492e49cfa5c15905703b71b507f"}
Oct 31 02:16:17.572997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:17.572873    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96" event={"ID":"e6538fad-9fb0-42e3-9e97-d54de4ebafc6","Type":"ContainerStarted","Data":"4a4ae3c1eb27a073b8c430b9b0c2d77d545694df4f45c2d2b807868a64f35640"}
Oct 31 02:16:17.589252 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:17.589210    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz" podStartSLOduration=2.589197638 podStartE2EDuration="2.589197638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:17.587185654 +0000 UTC m=+3335.535834046" watchObservedRunningTime="2026-10-31 02:16:17.589197638 +0000 UTC m=+3335.537846029"
Oct 31 02:16:17.666065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:17.653642    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96" podStartSLOduration=2.653617434 podStartE2EDuration="2.653617434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:17.603900945 +0000 UTC m=+3335.552549345" watchObservedRunningTime="2026-10-31 02:16:17.653617434 +0000 UTC m=+3335.602265834"
Oct 31 02:16:18.175565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:18.175530    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" path="/var/lib/kubelet/pods/c7e17ef2-7de5-45d9-85cd-a339307d8477/volumes"
Oct 31 02:16:18.581971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:18.581887    5293 generic.go:356] "Generic (PLEG): container finished" podID="e6538fad-9fb0-42e3-9e97-d54de4ebafc6" containerID="5d0bd0d9d05c508bc25683fb0eefc10eafae5492e49cfa5c15905703b71b507f" exitCode=0
Oct 31 02:16:18.582395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:18.581973    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96" event={"ID":"e6538fad-9fb0-42e3-9e97-d54de4ebafc6","Type":"ContainerDied","Data":"5d0bd0d9d05c508bc25683fb0eefc10eafae5492e49cfa5c15905703b71b507f"}
Oct 31 02:16:18.583978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:18.583885    5293 generic.go:356] "Generic (PLEG): container finished" podID="26eb0659-7c39-451d-8e0b-7c58b85908a8" containerID="a3b550cf42f6946b26a41753b6dcbd0b7e06385f98341174ade72b7e55c4a961" exitCode=0
Oct 31 02:16:18.583978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:18.583915    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz" event={"ID":"26eb0659-7c39-451d-8e0b-7c58b85908a8","Type":"ContainerDied","Data":"a3b550cf42f6946b26a41753b6dcbd0b7e06385f98341174ade72b7e55c4a961"}
Oct 31 02:16:20.030950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.030915    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:20.169569 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.169542    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:20.177741 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.177714    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-kube-api-access-fs282\" (UniqueName: \"kubernetes.io/projected/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-kube-api-access-fs282\") pod \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") "
Oct 31 02:16:20.177926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.177792    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-openshift-ca\") pod \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") "
Oct 31 02:16:20.177926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.177889    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-xks-ca\") pod \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\" (UID: \"e6538fad-9fb0-42e3-9e97-d54de4ebafc6\") "
Oct 31 02:16:20.178219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.178189    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-openshift-ca" pod "e6538fad-9fb0-42e3-9e97-d54de4ebafc6" (UID: "e6538fad-9fb0-42e3-9e97-d54de4ebafc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:20.178772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.178726    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:20.180245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.180223    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-kube-api-access-fs282" pod "e6538fad-9fb0-42e3-9e97-d54de4ebafc6" (UID: "e6538fad-9fb0-42e3-9e97-d54de4ebafc6"). InnerVolumeSpecName "kube-api-access-fs282". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:20.180488 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.180470    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-xks-ca" pod "e6538fad-9fb0-42e3-9e97-d54de4ebafc6" (UID: "e6538fad-9fb0-42e3-9e97-d54de4ebafc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:20.279889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.279804    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-openshift-ca\") pod \"26eb0659-7c39-451d-8e0b-7c58b85908a8\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") "
Oct 31 02:16:20.279889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.279853    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-xks-ca\") pod \"26eb0659-7c39-451d-8e0b-7c58b85908a8\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") "
Oct 31 02:16:20.280142 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.280118    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-openshift-ca" pod "26eb0659-7c39-451d-8e0b-7c58b85908a8" (UID: "26eb0659-7c39-451d-8e0b-7c58b85908a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:20.280199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.280128    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26eb0659-7c39-451d-8e0b-7c58b85908a8-kube-api-access-nmz5g\" (UniqueName: \"kubernetes.io/projected/26eb0659-7c39-451d-8e0b-7c58b85908a8-kube-api-access-nmz5g\") pod \"26eb0659-7c39-451d-8e0b-7c58b85908a8\" (UID: \"26eb0659-7c39-451d-8e0b-7c58b85908a8\") "
Oct 31 02:16:20.281684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.280805    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs282\" (UniqueName: \"kubernetes.io/projected/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-kube-api-access-fs282\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:20.281684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.280836    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6538fad-9fb0-42e3-9e97-d54de4ebafc6-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:20.281684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.280851    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:20.282273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.282248    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-xks-ca" pod "26eb0659-7c39-451d-8e0b-7c58b85908a8" (UID: "26eb0659-7c39-451d-8e0b-7c58b85908a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:20.282647 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.282630    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26eb0659-7c39-451d-8e0b-7c58b85908a8-kube-api-access-nmz5g" pod "26eb0659-7c39-451d-8e0b-7c58b85908a8" (UID: "26eb0659-7c39-451d-8e0b-7c58b85908a8"). InnerVolumeSpecName "kube-api-access-nmz5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:20.381927 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.381894    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmz5g\" (UniqueName: \"kubernetes.io/projected/26eb0659-7c39-451d-8e0b-7c58b85908a8-kube-api-access-nmz5g\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:20.381927 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.381921    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26eb0659-7c39-451d-8e0b-7c58b85908a8-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:20.687330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.687291    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz"
Oct 31 02:16:20.687573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.687316    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57403e6a-5pfsz" event={"ID":"26eb0659-7c39-451d-8e0b-7c58b85908a8","Type":"ContainerDied","Data":"8ef4a4d2bec8c8cb7ec9b79c4cb38195f8ab8dfbb29bff47a41a8cfc47229f80"}
Oct 31 02:16:20.687573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.687390    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ef4a4d2bec8c8cb7ec9b79c4cb38195f8ab8dfbb29bff47a41a8cfc47229f80"
Oct 31 02:16:20.689321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.689301    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96"
Oct 31 02:16:20.689458 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.689302    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e82121-kwc96" event={"ID":"e6538fad-9fb0-42e3-9e97-d54de4ebafc6","Type":"ContainerDied","Data":"4a4ae3c1eb27a073b8c430b9b0c2d77d545694df4f45c2d2b807868a64f35640"}
Oct 31 02:16:20.689458 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:20.689444    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a4ae3c1eb27a073b8c430b9b0c2d77d545694df4f45c2d2b807868a64f35640"
Oct 31 02:16:23.148724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:23.148635    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929" podUID="daae2c1d-05e9-4a5f-892a-2f454f4b1f76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:16:23.149154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:23.148762    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6" podUID="be9d1606-5c4a-4d16-8370-958320546e86" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:16:28.517657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:28.517621    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:16:31.924336 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:31.924107    5293 generic.go:356] "Generic (PLEG): container finished" podID="be9d1606-5c4a-4d16-8370-958320546e86" containerID="a95a3da163e34fd9c9813dad9d604445d588a90d9532d56358a0286023b07be5" exitCode=0
Oct 31 02:16:31.924336 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:31.924271    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6" event={"ID":"be9d1606-5c4a-4d16-8370-958320546e86","Type":"ContainerDied","Data":"a95a3da163e34fd9c9813dad9d604445d588a90d9532d56358a0286023b07be5"}
Oct 31 02:16:31.941239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:31.940879    5293 generic.go:356] "Generic (PLEG): container finished" podID="daae2c1d-05e9-4a5f-892a-2f454f4b1f76" containerID="ca0b1f4b59afd4f8563909be2d63776daa6c07f674ca54e579dbfc35cbef1c78" exitCode=0
Oct 31 02:16:31.941239 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:31.941151    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929" event={"ID":"daae2c1d-05e9-4a5f-892a-2f454f4b1f76","Type":"ContainerDied","Data":"ca0b1f4b59afd4f8563909be2d63776daa6c07f674ca54e579dbfc35cbef1c78"}
Oct 31 02:16:32.766600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.766572    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 02:16:32.783621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.783589    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-kube-api-access-wxdvn\" (UniqueName: \"kubernetes.io/projected/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-kube-api-access-wxdvn\") pod \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\" (UID: \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\") "
Oct 31 02:16:32.783772 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.783668    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-plugins-config-volume\") pod \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\" (UID: \"daae2c1d-05e9-4a5f-892a-2f454f4b1f76\") "
Oct 31 02:16:32.784377 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.784352    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-plugins-config-volume" pod "daae2c1d-05e9-4a5f-892a-2f454f4b1f76" (UID: "daae2c1d-05e9-4a5f-892a-2f454f4b1f76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:32.785876 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.785853    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-kube-api-access-wxdvn" pod "daae2c1d-05e9-4a5f-892a-2f454f4b1f76" (UID: "daae2c1d-05e9-4a5f-892a-2f454f4b1f76"). InnerVolumeSpecName "kube-api-access-wxdvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:32.824344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.824319    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 02:16:32.885199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.885117    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9d1606-5c4a-4d16-8370-958320546e86-kube-api-access-kqwfx\" (UniqueName: \"kubernetes.io/projected/be9d1606-5c4a-4d16-8370-958320546e86-kube-api-access-kqwfx\") pod \"be9d1606-5c4a-4d16-8370-958320546e86\" (UID: \"be9d1606-5c4a-4d16-8370-958320546e86\") "
Oct 31 02:16:32.885356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.885310    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9d1606-5c4a-4d16-8370-958320546e86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9d1606-5c4a-4d16-8370-958320546e86-plugins-config-volume\") pod \"be9d1606-5c4a-4d16-8370-958320546e86\" (UID: \"be9d1606-5c4a-4d16-8370-958320546e86\") "
Oct 31 02:16:32.885676 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.885651    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9d1606-5c4a-4d16-8370-958320546e86-plugins-config-volume" pod "be9d1606-5c4a-4d16-8370-958320546e86" (UID: "be9d1606-5c4a-4d16-8370-958320546e86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:32.885768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.885664    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxdvn\" (UniqueName: \"kubernetes.io/projected/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-kube-api-access-wxdvn\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:32.885768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.885704    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae2c1d-05e9-4a5f-892a-2f454f4b1f76-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:32.887527 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.887505    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9d1606-5c4a-4d16-8370-958320546e86-kube-api-access-kqwfx" pod "be9d1606-5c4a-4d16-8370-958320546e86" (UID: "be9d1606-5c4a-4d16-8370-958320546e86"). InnerVolumeSpecName "kube-api-access-kqwfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:32.964770 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.964741    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"
Oct 31 02:16:32.965294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.964773    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6" event={"ID":"be9d1606-5c4a-4d16-8370-958320546e86","Type":"ContainerDied","Data":"532647a816a88fe0cb4225a2a6511cf144e2fec567385eb1b5d07a862e04dcd3"}
Oct 31 02:16:32.965294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.964834    5293 scope.go:122] "RemoveContainer" containerID="a95a3da163e34fd9c9813dad9d604445d588a90d9532d56358a0286023b07be5"
Oct 31 02:16:32.966830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.966809    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929" event={"ID":"daae2c1d-05e9-4a5f-892a-2f454f4b1f76","Type":"ContainerDied","Data":"f1a332d8a79b5d900b8962eec3a3fea7f3a79553ed0a3e9983b4af38736f1633"}
Oct 31 02:16:32.966932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.966820    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"
Oct 31 02:16:32.976557 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.976538    5293 scope.go:122] "RemoveContainer" containerID="ca0b1f4b59afd4f8563909be2d63776daa6c07f674ca54e579dbfc35cbef1c78"
Oct 31 02:16:32.986843 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.986819    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9d1606-5c4a-4d16-8370-958320546e86-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:32.986940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:32.986847    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwfx\" (UniqueName: \"kubernetes.io/projected/be9d1606-5c4a-4d16-8370-958320546e86-kube-api-access-kqwfx\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:33.000954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.000930    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"]
Oct 31 02:16:33.004324 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.004283    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3cf6f4-79f5765c79-vwfr6"]
Oct 31 02:16:33.018285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.018251    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"]
Oct 31 02:16:33.025123 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.025089    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3cf6f4-c46fdf49f-cd929"]
Oct 31 02:16:33.338066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.337398    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"]
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.338972    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e6538fad-9fb0-42e3-9e97-d54de4ebafc6" containerName="revoke-keys"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.338998    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6538fad-9fb0-42e3-9e97-d54de4ebafc6" containerName="revoke-keys"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.339013    5293 cpu_manager.go:414] "containerMap: removing container" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.339022    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.339066    5293 cpu_manager.go:414] "containerMap: removing container" podUID="daae2c1d-05e9-4a5f-892a-2f454f4b1f76" containerName="payload-processing"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.339075    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daae2c1d-05e9-4a5f-892a-2f454f4b1f76" containerName="payload-processing"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.339105    5293 cpu_manager.go:414] "containerMap: removing container" podUID="26eb0659-7c39-451d-8e0b-7c58b85908a8" containerName="revoke-keys"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.339113    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26eb0659-7c39-451d-8e0b-7c58b85908a8" containerName="revoke-keys"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.339134    5293 cpu_manager.go:414] "containerMap: removing container" podUID="be9d1606-5c4a-4d16-8370-958320546e86" containerName="payload-pre-processing"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.339141    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9d1606-5c4a-4d16-8370-958320546e86" containerName="payload-pre-processing"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.340773    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9d1606-5c4a-4d16-8370-958320546e86" containerName="payload-pre-processing"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.340815    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6538fad-9fb0-42e3-9e97-d54de4ebafc6" containerName="revoke-keys"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.340827    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e17ef2-7de5-45d9-85cd-a339307d8477" containerName="main"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.340841    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="daae2c1d-05e9-4a5f-892a-2f454f4b1f76" containerName="payload-processing"
Oct 31 02:16:33.341117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.340857    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="26eb0659-7c39-451d-8e0b-7c58b85908a8" containerName="revoke-keys"
Oct 31 02:16:33.426069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.417568    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.426069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.423881    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"]
Oct 31 02:16:33.458064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.430057    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:16:33.458064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.430602    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:16:33.479941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.479904    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.480144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.480011    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.480144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.480099    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5st\" (UniqueName: \"kubernetes.io/projected/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-kube-api-access-9g5st\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.581461 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.581419    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.581981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.581523    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.581981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.581553    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5st\" (UniqueName: \"kubernetes.io/projected/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-kube-api-access-9g5st\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.582421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.582379    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.584028 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.584011    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.593306 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.593236    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5st\" (UniqueName: \"kubernetes.io/projected/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-kube-api-access-9g5st\") pod \"maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:33.787983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:33.787952    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:34.115153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:34.115115    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"]
Oct 31 02:16:34.118560 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:16:34.118524    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa4c788_33fe_422c_a58b_e87a0f59f4a7.slice/crio-1c1d396d0c23f9a988eb60d552cc24e21f6e47663a14bf08237a45b7204161e6 WatchSource:0}: Error finding container 1c1d396d0c23f9a988eb60d552cc24e21f6e47663a14bf08237a45b7204161e6: Status 404 returned error can't find the container with id 1c1d396d0c23f9a988eb60d552cc24e21f6e47663a14bf08237a45b7204161e6
Oct 31 02:16:34.175693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:34.175666    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9d1606-5c4a-4d16-8370-958320546e86" path="/var/lib/kubelet/pods/be9d1606-5c4a-4d16-8370-958320546e86/volumes"
Oct 31 02:16:34.176121 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:34.176106    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daae2c1d-05e9-4a5f-892a-2f454f4b1f76" path="/var/lib/kubelet/pods/daae2c1d-05e9-4a5f-892a-2f454f4b1f76/volumes"
Oct 31 02:16:35.012155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:35.011556    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8" event={"ID":"1fa4c788-33fe-422c-a58b-e87a0f59f4a7","Type":"ContainerStarted","Data":"cadc5fb0e43b8d5857b2257a3ffbf28f8f7b174a35a516977d6c8bd51905f1c9"}
Oct 31 02:16:35.012155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:35.011625    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8" event={"ID":"1fa4c788-33fe-422c-a58b-e87a0f59f4a7","Type":"ContainerStarted","Data":"1c1d396d0c23f9a988eb60d552cc24e21f6e47663a14bf08237a45b7204161e6"}
Oct 31 02:16:35.022229 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:35.022198    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"]
Oct 31 02:16:35.028343 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:35.028305    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3cf6f4-fgghx"]
Oct 31 02:16:36.023203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.023168    5293 generic.go:356] "Generic (PLEG): container finished" podID="1fa4c788-33fe-422c-a58b-e87a0f59f4a7" containerID="cadc5fb0e43b8d5857b2257a3ffbf28f8f7b174a35a516977d6c8bd51905f1c9" exitCode=0
Oct 31 02:16:36.023628 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.023208    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8" event={"ID":"1fa4c788-33fe-422c-a58b-e87a0f59f4a7","Type":"ContainerDied","Data":"cadc5fb0e43b8d5857b2257a3ffbf28f8f7b174a35a516977d6c8bd51905f1c9"}
Oct 31 02:16:36.054288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.054259    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"]
Oct 31 02:16:36.058780 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.058757    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.066218 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.066189    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"]
Oct 31 02:16:36.148765 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.148732    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.148945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.148867    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ff6c\" (UniqueName: \"kubernetes.io/projected/042bc481-4c8e-4a47-b767-09b3a10716e4-kube-api-access-2ff6c\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.149105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.149084    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.174735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.174707    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c92982-f897-41c1-b377-93c8cb7d55d1" path="/var/lib/kubelet/pods/95c92982-f897-41c1-b377-93c8cb7d55d1/volumes"
Oct 31 02:16:36.250089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.250052    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.250263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.250152    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ff6c\" (UniqueName: \"kubernetes.io/projected/042bc481-4c8e-4a47-b767-09b3a10716e4-kube-api-access-2ff6c\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.250331 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.250315    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.250946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.250921    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.252568 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.252545    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.263034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.263009    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ff6c\" (UniqueName: \"kubernetes.io/projected/042bc481-4c8e-4a47-b767-09b3a10716e4-kube-api-access-2ff6c\") pod \"maas-api-revoke-sub-fae3dbe7-w9lns\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.388947 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.388862    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:36.719267 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:36.719226    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"]
Oct 31 02:16:36.720147 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:16:36.720114    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042bc481_4c8e_4a47_b767_09b3a10716e4.slice/crio-1446da123b0968ec768647b1a38e597962ab842220053e6bdaffa15915c2dd26 WatchSource:0}: Error finding container 1446da123b0968ec768647b1a38e597962ab842220053e6bdaffa15915c2dd26: Status 404 returned error can't find the container with id 1446da123b0968ec768647b1a38e597962ab842220053e6bdaffa15915c2dd26
Oct 31 02:16:37.034889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.034840    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns" event={"ID":"042bc481-4c8e-4a47-b767-09b3a10716e4","Type":"ContainerStarted","Data":"65d0c0d95f9223c17ffda9ff4dd23b2dd9bd7a123131534c2a0000fcde1acdf9"}
Oct 31 02:16:37.035340 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.034913    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns" event={"ID":"042bc481-4c8e-4a47-b767-09b3a10716e4","Type":"ContainerStarted","Data":"1446da123b0968ec768647b1a38e597962ab842220053e6bdaffa15915c2dd26"}
Oct 31 02:16:37.184792 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.184764    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:37.207618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.207428    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns" podStartSLOduration=1.207408765 podStartE2EDuration="1.207408765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:37.061193055 +0000 UTC m=+3355.009841448" watchObservedRunningTime="2026-10-31 02:16:37.207408765 +0000 UTC m=+3355.156057156"
Oct 31 02:16:37.268301 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.268268    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-openshift-ca\") pod \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") "
Oct 31 02:16:37.268438 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.268307    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-xks-ca\") pod \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") "
Oct 31 02:16:37.268438 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.268393    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-kube-api-access-9g5st\" (UniqueName: \"kubernetes.io/projected/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-kube-api-access-9g5st\") pod \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\" (UID: \"1fa4c788-33fe-422c-a58b-e87a0f59f4a7\") "
Oct 31 02:16:37.268707 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.268662    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-openshift-ca" pod "1fa4c788-33fe-422c-a58b-e87a0f59f4a7" (UID: "1fa4c788-33fe-422c-a58b-e87a0f59f4a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:37.268896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.268876    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:37.271338 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.271310    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-kube-api-access-9g5st" pod "1fa4c788-33fe-422c-a58b-e87a0f59f4a7" (UID: "1fa4c788-33fe-422c-a58b-e87a0f59f4a7"). InnerVolumeSpecName "kube-api-access-9g5st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:37.271452 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.271377    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-xks-ca" pod "1fa4c788-33fe-422c-a58b-e87a0f59f4a7" (UID: "1fa4c788-33fe-422c-a58b-e87a0f59f4a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:37.370744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.370714    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:37.370744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:37.370742    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g5st\" (UniqueName: \"kubernetes.io/projected/1fa4c788-33fe-422c-a58b-e87a0f59f4a7-kube-api-access-9g5st\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:38.045084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:38.045031    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"
Oct 31 02:16:38.045084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:38.045065    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8" event={"ID":"1fa4c788-33fe-422c-a58b-e87a0f59f4a7","Type":"ContainerDied","Data":"1c1d396d0c23f9a988eb60d552cc24e21f6e47663a14bf08237a45b7204161e6"}
Oct 31 02:16:38.045570 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:38.045149    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1d396d0c23f9a988eb60d552cc24e21f6e47663a14bf08237a45b7204161e6"
Oct 31 02:16:38.047009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:38.046983    5293 generic.go:356] "Generic (PLEG): container finished" podID="042bc481-4c8e-4a47-b767-09b3a10716e4" containerID="65d0c0d95f9223c17ffda9ff4dd23b2dd9bd7a123131534c2a0000fcde1acdf9" exitCode=0
Oct 31 02:16:38.047157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:38.047075    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns" event={"ID":"042bc481-4c8e-4a47-b767-09b3a10716e4","Type":"ContainerDied","Data":"65d0c0d95f9223c17ffda9ff4dd23b2dd9bd7a123131534c2a0000fcde1acdf9"}
Oct 31 02:16:39.223708 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.223677    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:39.249276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.248872    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-xks-ca\") pod \"042bc481-4c8e-4a47-b767-09b3a10716e4\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") "
Oct 31 02:16:39.249276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.248982    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042bc481-4c8e-4a47-b767-09b3a10716e4-kube-api-access-2ff6c\" (UniqueName: \"kubernetes.io/projected/042bc481-4c8e-4a47-b767-09b3a10716e4-kube-api-access-2ff6c\") pod \"042bc481-4c8e-4a47-b767-09b3a10716e4\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") "
Oct 31 02:16:39.249276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.249121    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-openshift-ca\") pod \"042bc481-4c8e-4a47-b767-09b3a10716e4\" (UID: \"042bc481-4c8e-4a47-b767-09b3a10716e4\") "
Oct 31 02:16:39.250837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.250810    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-openshift-ca" pod "042bc481-4c8e-4a47-b767-09b3a10716e4" (UID: "042bc481-4c8e-4a47-b767-09b3a10716e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:39.252174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.252149    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-xks-ca" pod "042bc481-4c8e-4a47-b767-09b3a10716e4" (UID: "042bc481-4c8e-4a47-b767-09b3a10716e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:39.254627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.254600    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042bc481-4c8e-4a47-b767-09b3a10716e4-kube-api-access-2ff6c" pod "042bc481-4c8e-4a47-b767-09b3a10716e4" (UID: "042bc481-4c8e-4a47-b767-09b3a10716e4"). InnerVolumeSpecName "kube-api-access-2ff6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:39.352157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.352083    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:39.352157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.352112    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ff6c\" (UniqueName: \"kubernetes.io/projected/042bc481-4c8e-4a47-b767-09b3a10716e4-kube-api-access-2ff6c\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:39.352157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.352122    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042bc481-4c8e-4a47-b767-09b3a10716e4-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:39.883398 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.883360    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"]
Oct 31 02:16:39.884371 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.884344    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1fa4c788-33fe-422c-a58b-e87a0f59f4a7" containerName="revoke-keys"
Oct 31 02:16:39.884371 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.884369    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa4c788-33fe-422c-a58b-e87a0f59f4a7" containerName="revoke-keys"
Oct 31 02:16:39.884600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.884398    5293 cpu_manager.go:414] "containerMap: removing container" podUID="042bc481-4c8e-4a47-b767-09b3a10716e4" containerName="revoke-keys"
Oct 31 02:16:39.884600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.884407    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042bc481-4c8e-4a47-b767-09b3a10716e4" containerName="revoke-keys"
Oct 31 02:16:39.884600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.884554    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa4c788-33fe-422c-a58b-e87a0f59f4a7" containerName="revoke-keys"
Oct 31 02:16:39.884600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.884572    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="042bc481-4c8e-4a47-b767-09b3a10716e4" containerName="revoke-keys"
Oct 31 02:16:39.895895 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.895430    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:39.916130 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.916105    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"]
Oct 31 02:16:39.965589 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.965541    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:39.965834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.965807    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74bgp\" (UniqueName: \"kubernetes.io/projected/b3367513-7f0c-4daf-a1d3-3c8893feeab2-kube-api-access-74bgp\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:39.965925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:39.965906    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:40.063857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.063823    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns"
Oct 31 02:16:40.063857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.063849    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3dbe7-w9lns" event={"ID":"042bc481-4c8e-4a47-b767-09b3a10716e4","Type":"ContainerDied","Data":"1446da123b0968ec768647b1a38e597962ab842220053e6bdaffa15915c2dd26"}
Oct 31 02:16:40.064058 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.063895    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1446da123b0968ec768647b1a38e597962ab842220053e6bdaffa15915c2dd26"
Oct 31 02:16:40.066875 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.066843    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:40.067060 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.067024    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74bgp\" (UniqueName: \"kubernetes.io/projected/b3367513-7f0c-4daf-a1d3-3c8893feeab2-kube-api-access-74bgp\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:40.067138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.067125    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:40.067795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.067772    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:40.069619 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.069593    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:40.082970 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.082818    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74bgp\" (UniqueName: \"kubernetes.io/projected/b3367513-7f0c-4daf-a1d3-3c8893feeab2-kube-api-access-74bgp\") pod \"maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:40.212648 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.212618    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:40.517862 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:40.517788    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"]
Oct 31 02:16:40.518968 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:16:40.518936    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3367513_7f0c_4daf_a1d3_3c8893feeab2.slice/crio-1278ef0a4bc14f9e0dc0a41c42ee6d43398fd54f231d2d03eca3f1d0ef284de4 WatchSource:0}: Error finding container 1278ef0a4bc14f9e0dc0a41c42ee6d43398fd54f231d2d03eca3f1d0ef284de4: Status 404 returned error can't find the container with id 1278ef0a4bc14f9e0dc0a41c42ee6d43398fd54f231d2d03eca3f1d0ef284de4
Oct 31 02:16:41.026590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:41.026554    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"]
Oct 31 02:16:41.030071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:41.030035    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564e1078-4xc57"]
Oct 31 02:16:41.073247 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:41.073211    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b" event={"ID":"b3367513-7f0c-4daf-a1d3-3c8893feeab2","Type":"ContainerStarted","Data":"182201c1f366473905829e9e42c5e843c751bed78cf2794ee647f50cedfcfa45"}
Oct 31 02:16:41.073367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:41.073263    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b" event={"ID":"b3367513-7f0c-4daf-a1d3-3c8893feeab2","Type":"ContainerStarted","Data":"1278ef0a4bc14f9e0dc0a41c42ee6d43398fd54f231d2d03eca3f1d0ef284de4"}
Oct 31 02:16:41.094031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:41.093984    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b" podStartSLOduration=2.093967337 podStartE2EDuration="2.093967337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:41.091814087 +0000 UTC m=+3359.040462480" watchObservedRunningTime="2026-10-31 02:16:41.093967337 +0000 UTC m=+3359.042615730"
Oct 31 02:16:41.522844 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:16:41.522814    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3367513_7f0c_4daf_a1d3_3c8893feeab2.slice/crio-182201c1f366473905829e9e42c5e843c751bed78cf2794ee647f50cedfcfa45.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 02:16:41.523264 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:16:41.522909    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3367513_7f0c_4daf_a1d3_3c8893feeab2.slice/crio-conmon-182201c1f366473905829e9e42c5e843c751bed78cf2794ee647f50cedfcfa45.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 02:16:41.780397 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:16:41.780311    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:16:42.082262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:42.082180    5293 generic.go:356] "Generic (PLEG): container finished" podID="b3367513-7f0c-4daf-a1d3-3c8893feeab2" containerID="182201c1f366473905829e9e42c5e843c751bed78cf2794ee647f50cedfcfa45" exitCode=0
Oct 31 02:16:42.082262 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:42.082232    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b" event={"ID":"b3367513-7f0c-4daf-a1d3-3c8893feeab2","Type":"ContainerDied","Data":"182201c1f366473905829e9e42c5e843c751bed78cf2794ee647f50cedfcfa45"}
Oct 31 02:16:42.175469 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:42.175440    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb2ecbb7-d38b-4115-8a8b-48663b9c733a" path="/var/lib/kubelet/pods/eb2ecbb7-d38b-4115-8a8b-48663b9c733a/volumes"
Oct 31 02:16:42.692069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:42.691186    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"]
Oct 31 02:16:42.692069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:42.691455    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy" containerID="cri-o://a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a" gracePeriod=30
Oct 31 02:16:42.708573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:42.701189    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"]
Oct 31 02:16:42.708573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:42.701410    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy" containerID="cri-o://1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a" gracePeriod=30
Oct 31 02:16:43.404950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.404921    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:43.531909 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.531851    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-xks-ca\") pod \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") "
Oct 31 02:16:43.532168 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.532051    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3367513-7f0c-4daf-a1d3-3c8893feeab2-kube-api-access-74bgp\" (UniqueName: \"kubernetes.io/projected/b3367513-7f0c-4daf-a1d3-3c8893feeab2-kube-api-access-74bgp\") pod \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") "
Oct 31 02:16:43.536105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.536027    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-xks-ca" pod "b3367513-7f0c-4daf-a1d3-3c8893feeab2" (UID: "b3367513-7f0c-4daf-a1d3-3c8893feeab2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:43.536714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.532324    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-openshift-ca\") pod \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\" (UID: \"b3367513-7f0c-4daf-a1d3-3c8893feeab2\") "
Oct 31 02:16:43.536714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.536200    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-openshift-ca" pod "b3367513-7f0c-4daf-a1d3-3c8893feeab2" (UID: "b3367513-7f0c-4daf-a1d3-3c8893feeab2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:43.537636 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.537611    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:43.537753 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.537656    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3367513-7f0c-4daf-a1d3-3c8893feeab2-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:43.550367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.550108    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3367513-7f0c-4daf-a1d3-3c8893feeab2-kube-api-access-74bgp" pod "b3367513-7f0c-4daf-a1d3-3c8893feeab2" (UID: "b3367513-7f0c-4daf-a1d3-3c8893feeab2"). InnerVolumeSpecName "kube-api-access-74bgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:43.639966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:43.639869    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74bgp\" (UniqueName: \"kubernetes.io/projected/b3367513-7f0c-4daf-a1d3-3c8893feeab2-kube-api-access-74bgp\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:44.128063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:44.122222    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b" event={"ID":"b3367513-7f0c-4daf-a1d3-3c8893feeab2","Type":"ContainerDied","Data":"1278ef0a4bc14f9e0dc0a41c42ee6d43398fd54f231d2d03eca3f1d0ef284de4"}
Oct 31 02:16:44.128063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:44.122292    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1278ef0a4bc14f9e0dc0a41c42ee6d43398fd54f231d2d03eca3f1d0ef284de4"
Oct 31 02:16:44.128063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:44.122373    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"
Oct 31 02:16:45.723573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:45.723537    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"]
Oct 31 02:16:45.729092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:45.729062    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-20538a-lchf8"]
Oct 31 02:16:46.175266 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.175233    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa4c788-33fe-422c-a58b-e87a0f59f4a7" path="/var/lib/kubelet/pods/1fa4c788-33fe-422c-a58b-e87a0f59f4a7/volumes"
Oct 31 02:16:46.651514 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.651474    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"]
Oct 31 02:16:46.652322 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.652298    5293 cpu_manager.go:414] "containerMap: removing container" podUID="b3367513-7f0c-4daf-a1d3-3c8893feeab2" containerName="revoke-keys"
Oct 31 02:16:46.652322 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.652320    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3367513-7f0c-4daf-a1d3-3c8893feeab2" containerName="revoke-keys"
Oct 31 02:16:46.652522 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.652490    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3367513-7f0c-4daf-a1d3-3c8893feeab2" containerName="revoke-keys"
Oct 31 02:16:46.656172 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.656151    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.660369 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.660346    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:16:46.663135 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.660999    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:16:46.666958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.666935    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"]
Oct 31 02:16:46.733288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.733238    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.733744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.733467    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.733744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.733732    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvj2f\" (UniqueName: \"kubernetes.io/projected/044d45ed-06e4-41a3-bb44-bea63b2db1e7-kube-api-access-nvj2f\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.834831 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.834786    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvj2f\" (UniqueName: \"kubernetes.io/projected/044d45ed-06e4-41a3-bb44-bea63b2db1e7-kube-api-access-nvj2f\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.835059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.834858    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.835059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.834988    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.835966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.835936    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.837915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.837892    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.847098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.846950    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvj2f\" (UniqueName: \"kubernetes.io/projected/044d45ed-06e4-41a3-bb44-bea63b2db1e7-kube-api-access-nvj2f\") pod \"maas-api-revoke-sub-aa465b6e-znnsx\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:46.977400 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:46.977313    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:47.270801 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:47.270767    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"]
Oct 31 02:16:47.271640 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:16:47.271616    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044d45ed_06e4_41a3_bb44_bea63b2db1e7.slice/crio-ef1960670788e185fa9d4cae0c95565a39e736e4df3bc596e8517cd51778c25a WatchSource:0}: Error finding container ef1960670788e185fa9d4cae0c95565a39e736e4df3bc596e8517cd51778c25a: Status 404 returned error can't find the container with id ef1960670788e185fa9d4cae0c95565a39e736e4df3bc596e8517cd51778c25a
Oct 31 02:16:48.157434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:48.157395    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx" event={"ID":"044d45ed-06e4-41a3-bb44-bea63b2db1e7","Type":"ContainerStarted","Data":"70eb146363d91ddf8ccd4c1618ec58685eba6776d88cee2cf65e5380e75908fb"}
Oct 31 02:16:48.157959 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:48.157462    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx" event={"ID":"044d45ed-06e4-41a3-bb44-bea63b2db1e7","Type":"ContainerStarted","Data":"ef1960670788e185fa9d4cae0c95565a39e736e4df3bc596e8517cd51778c25a"}
Oct 31 02:16:48.178371 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:48.178291    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx" podStartSLOduration=2.178272517 podStartE2EDuration="2.178272517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:48.176005615 +0000 UTC m=+3366.124654008" watchObservedRunningTime="2026-10-31 02:16:48.178272517 +0000 UTC m=+3366.126920910"
Oct 31 02:16:49.104589 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.104559    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 02:16:49.168800 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.168777    5293 generic.go:356] "Generic (PLEG): container finished" podID="044d45ed-06e4-41a3-bb44-bea63b2db1e7" containerID="70eb146363d91ddf8ccd4c1618ec58685eba6776d88cee2cf65e5380e75908fb" exitCode=0
Oct 31 02:16:49.169181 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.168864    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx" event={"ID":"044d45ed-06e4-41a3-bb44-bea63b2db1e7","Type":"ContainerDied","Data":"70eb146363d91ddf8ccd4c1618ec58685eba6776d88cee2cf65e5380e75908fb"}
Oct 31 02:16:49.170511 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.170491    5293 generic.go:356] "Generic (PLEG): container finished" podID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerID="a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a" exitCode=0
Oct 31 02:16:49.170639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.170535    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"
Oct 31 02:16:49.170639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.170570    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" event={"ID":"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd","Type":"ContainerDied","Data":"a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a"}
Oct 31 02:16:49.170639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.170616    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd" event={"ID":"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd","Type":"ContainerDied","Data":"2b9e2e2029718a071c85b6d6d1049eca5fe6747d589cc7aa3aecfdf07d1da04d"}
Oct 31 02:16:49.170639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.170637    5293 scope.go:122] "RemoveContainer" containerID="a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a"
Oct 31 02:16:49.174606 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.174579    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-socket\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.174722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.174653    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-certs\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.174722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.174698    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-credential-socket\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.174841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.174786    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-data\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.174841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.174817    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-token\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-token\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.174948 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.174900    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-socket" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.175004 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.174951    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-podinfo\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.175004 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.174983    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-certs" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.175132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175005    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-kube-api-access-29q5g\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-kube-api-access-29q5g\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.175132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175054    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-envoy\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.175132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175084    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istiod-ca-cert\") pod \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\" (UID: \"a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd\") "
Oct 31 02:16:49.175282 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175178    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-data" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.175282 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175212    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-credential-socket" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.175827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175803    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.175910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175833    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.175910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175848    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.175910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.175862    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.176254 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.176232    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istiod-ca-cert" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:49.178093 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.178021    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-token" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:49.178279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.178250    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-podinfo" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:16:49.178661 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.178627    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-envoy" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.179814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.179788    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-kube-api-access-29q5g" pod "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" (UID: "a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd"). InnerVolumeSpecName "kube-api-access-29q5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:49.234535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.234510    5293 scope.go:122] "RemoveContainer" containerID="a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a"
Oct 31 02:16:49.237421 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:16:49.237390    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a\": container with ID starting with a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a not found: ID does not exist" containerID="a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a"
Oct 31 02:16:49.237557 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.237432    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a"} err="failed to get container status \"a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a\": rpc error: code = NotFound desc = could not find container \"a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a\": container with ID starting with a2e453d7624a15b87dc11b44d684d111f0e3a923d0a9cede8b07a1eefbb58d8a not found: ID does not exist"
Oct 31 02:16:49.277370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.277336    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.277370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.277370    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.277627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.277388    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29q5g\" (UniqueName: \"kubernetes.io/projected/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-kube-api-access-29q5g\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.277627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.277404    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.277627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.277419    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.395714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.395685    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 02:16:49.479816 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.479777    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a38bc9-35da-45db-b4c4-18a32e1b5041-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a38bc9-35da-45db-b4c4-18a32e1b5041-istiod-ca-cert\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.479953 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.479921    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-kube-api-access-vhvn5\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-kube-api-access-vhvn5\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.480008 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.479962    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-token\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-token\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.480097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.480082    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-podinfo\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.480733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.480660    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-envoy\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.480917 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.480786    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a38bc9-35da-45db-b4c4-18a32e1b5041-istiod-ca-cert" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:49.481901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.481303    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-credential-socket\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.481901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.481360    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-data\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.481901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.481438    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-certs\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.481901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.481475    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-socket\") pod \"75a38bc9-35da-45db-b4c4-18a32e1b5041\" (UID: \"75a38bc9-35da-45db-b4c4-18a32e1b5041\") "
Oct 31 02:16:49.481901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.481705    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-credential-socket" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.482494 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.482272    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.482494 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.482299    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a38bc9-35da-45db-b4c4-18a32e1b5041-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.482494 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.482323    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-socket" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.482943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.482785    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-data" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.482943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.482935    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-podinfo" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:16:49.483181 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.483160    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-certs" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.483550 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.483525    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-kube-api-access-vhvn5" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "kube-api-access-vhvn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:49.483695 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.483670    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-token" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:49.484907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.484883    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-envoy" pod "75a38bc9-35da-45db-b4c4-18a32e1b5041" (UID: "75a38bc9-35da-45db-b4c4-18a32e1b5041"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.497092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.497070    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"]
Oct 31 02:16:49.503638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.503618    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-gz4xd"]
Oct 31 02:16:49.602063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.593240    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.602063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.593279    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.602063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.593294    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.602063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.593306    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.602063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.593319    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a38bc9-35da-45db-b4c4-18a32e1b5041-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.602063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.593332    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhvn5\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-kube-api-access-vhvn5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.602063 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:49.593345    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a38bc9-35da-45db-b4c4-18a32e1b5041-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:50.020792 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.020754    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"]
Oct 31 02:16:50.036798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.036763    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"]
Oct 31 02:16:50.177178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.177140    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" path="/var/lib/kubelet/pods/a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd/volumes"
Oct 31 02:16:50.180263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.180233    5293 generic.go:356] "Generic (PLEG): container finished" podID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerID="1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a" exitCode=0
Oct 31 02:16:50.180424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.180299    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"
Oct 31 02:16:50.180424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.180321    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" event={"ID":"75a38bc9-35da-45db-b4c4-18a32e1b5041","Type":"ContainerDied","Data":"1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a"}
Oct 31 02:16:50.180424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.180371    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9" event={"ID":"75a38bc9-35da-45db-b4c4-18a32e1b5041","Type":"ContainerDied","Data":"fd30c54257a1c1b2ccb83b20cd86fd2cc43826334170ea97ba3c1cc1ff5a2f04"}
Oct 31 02:16:50.180424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.180403    5293 scope.go:122] "RemoveContainer" containerID="1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a"
Oct 31 02:16:50.193506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.193484    5293 scope.go:122] "RemoveContainer" containerID="1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a"
Oct 31 02:16:50.193782 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:16:50.193759    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a\": container with ID starting with 1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a not found: ID does not exist" containerID="1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a"
Oct 31 02:16:50.193862 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.193794    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a"} err="failed to get container status \"1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a\": rpc error: code = NotFound desc = could not find container \"1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a\": container with ID starting with 1fd681561fb09b8802175a1444e26f468e6163c5c153397fb77fd4779541da6a not found: ID does not exist"
Oct 31 02:16:50.210563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.210537    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"]
Oct 31 02:16:50.214613 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.214582    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-xwrd9"]
Oct 31 02:16:50.553383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.553356    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:50.640144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.640111    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-xks-ca\") pod \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") "
Oct 31 02:16:50.640332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.640154    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-openshift-ca\") pod \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") "
Oct 31 02:16:50.640332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.640245    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044d45ed-06e4-41a3-bb44-bea63b2db1e7-kube-api-access-nvj2f\" (UniqueName: \"kubernetes.io/projected/044d45ed-06e4-41a3-bb44-bea63b2db1e7-kube-api-access-nvj2f\") pod \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\" (UID: \"044d45ed-06e4-41a3-bb44-bea63b2db1e7\") "
Oct 31 02:16:50.640650 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.640613    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-openshift-ca" pod "044d45ed-06e4-41a3-bb44-bea63b2db1e7" (UID: "044d45ed-06e4-41a3-bb44-bea63b2db1e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:50.640838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.640815    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:50.643030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.643001    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044d45ed-06e4-41a3-bb44-bea63b2db1e7-kube-api-access-nvj2f" pod "044d45ed-06e4-41a3-bb44-bea63b2db1e7" (UID: "044d45ed-06e4-41a3-bb44-bea63b2db1e7"). InnerVolumeSpecName "kube-api-access-nvj2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:50.643030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.643008    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-xks-ca" pod "044d45ed-06e4-41a3-bb44-bea63b2db1e7" (UID: "044d45ed-06e4-41a3-bb44-bea63b2db1e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:50.742434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.742403    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044d45ed-06e4-41a3-bb44-bea63b2db1e7-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:50.742541 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:50.742441    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvj2f\" (UniqueName: \"kubernetes.io/projected/044d45ed-06e4-41a3-bb44-bea63b2db1e7-kube-api-access-nvj2f\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:51.189896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:51.189864    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx"
Oct 31 02:16:51.190344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:51.189902    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa465b6e-znnsx" event={"ID":"044d45ed-06e4-41a3-bb44-bea63b2db1e7","Type":"ContainerDied","Data":"ef1960670788e185fa9d4cae0c95565a39e736e4df3bc596e8517cd51778c25a"}
Oct 31 02:16:51.190344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:51.189956    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef1960670788e185fa9d4cae0c95565a39e736e4df3bc596e8517cd51778c25a"
Oct 31 02:16:52.177184 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:52.177154    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" path="/var/lib/kubelet/pods/75a38bc9-35da-45db-b4c4-18a32e1b5041/volumes"
Oct 31 02:16:54.128287 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.128248    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"]
Oct 31 02:16:54.129105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129084    5293 cpu_manager.go:414] "containerMap: removing container" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy"
Oct 31 02:16:54.129105 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129106    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy"
Oct 31 02:16:54.129285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129127    5293 cpu_manager.go:414] "containerMap: removing container" podUID="044d45ed-06e4-41a3-bb44-bea63b2db1e7" containerName="revoke-keys"
Oct 31 02:16:54.129285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129134    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044d45ed-06e4-41a3-bb44-bea63b2db1e7" containerName="revoke-keys"
Oct 31 02:16:54.129285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129169    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy"
Oct 31 02:16:54.129285 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129177    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy"
Oct 31 02:16:54.129499 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129330    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ba9c46-10ea-43e1-9cbb-fee917d0b0cd" containerName="istio-proxy"
Oct 31 02:16:54.129499 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129357    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a38bc9-35da-45db-b4c4-18a32e1b5041" containerName="istio-proxy"
Oct 31 02:16:54.129499 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.129368    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="044d45ed-06e4-41a3-bb44-bea63b2db1e7" containerName="revoke-keys"
Oct 31 02:16:54.134683 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.134663    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.142898 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.142864    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"]
Oct 31 02:16:54.181462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.181435    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:16:54.181689 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.181599    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:16:54.206872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.206840    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.207164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.207025    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwh2\" (UniqueName: \"kubernetes.io/projected/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-kube-api-access-ftwh2\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.207164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.207145    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.328798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.327639    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwh2\" (UniqueName: \"kubernetes.io/projected/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-kube-api-access-ftwh2\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.328798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.328775    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.333067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.329718    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.335586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.335556    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.358431 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.358404    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.366575 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.366545    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwh2\" (UniqueName: \"kubernetes.io/projected/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-kube-api-access-ftwh2\") pod \"maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.453699 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.453668    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:54.748251 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:54.748116    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"]
Oct 31 02:16:55.021283 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:55.021242    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5" podUID="3419e3d8-bec8-4eba-9e85-dd542ed634dc" containerName="payload-processing" containerID="cri-o://5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11" gracePeriod=25
Oct 31 02:16:55.037725 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:55.037691    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6" podUID="eb03a89a-bd26-436c-9669-9a289b79dad1" containerName="payload-pre-processing" containerID="cri-o://8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d" gracePeriod=25
Oct 31 02:16:55.224176 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:55.224136    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx" event={"ID":"492e1026-b99a-4e7b-a65a-a89fbfb6dac2","Type":"ContainerStarted","Data":"6e65e191b4fff32a479ded81b3eb6bf5f965b44cad1879e6c9b2ade78ac4b936"}
Oct 31 02:16:55.224571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:55.224201    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx" event={"ID":"492e1026-b99a-4e7b-a65a-a89fbfb6dac2","Type":"ContainerStarted","Data":"9e8d1c5b9bf317952be3c62fba4807bf3f8ced5fef55283a276a1b945146d186"}
Oct 31 02:16:55.243116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:55.243070    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx" podStartSLOduration=1.243057279 podStartE2EDuration="1.243057279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:55.240528239 +0000 UTC m=+3373.189176644" watchObservedRunningTime="2026-10-31 02:16:55.243057279 +0000 UTC m=+3373.191705661"
Oct 31 02:16:56.236525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:56.236488    5293 generic.go:356] "Generic (PLEG): container finished" podID="492e1026-b99a-4e7b-a65a-a89fbfb6dac2" containerID="6e65e191b4fff32a479ded81b3eb6bf5f965b44cad1879e6c9b2ade78ac4b936" exitCode=0
Oct 31 02:16:56.236983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:56.236576    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx" event={"ID":"492e1026-b99a-4e7b-a65a-a89fbfb6dac2","Type":"ContainerDied","Data":"6e65e191b4fff32a479ded81b3eb6bf5f965b44cad1879e6c9b2ade78ac4b936"}
Oct 31 02:16:57.402344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.402323    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:57.508153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.508075    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-openshift-ca\") pod \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") "
Oct 31 02:16:57.508302 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.508148    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-xks-ca\") pod \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") "
Oct 31 02:16:57.508374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.508344    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-kube-api-access-ftwh2\" (UniqueName: \"kubernetes.io/projected/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-kube-api-access-ftwh2\") pod \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\" (UID: \"492e1026-b99a-4e7b-a65a-a89fbfb6dac2\") "
Oct 31 02:16:57.508459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.508440    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-openshift-ca" pod "492e1026-b99a-4e7b-a65a-a89fbfb6dac2" (UID: "492e1026-b99a-4e7b-a65a-a89fbfb6dac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:57.508966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.508945    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:57.510504 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.510477    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-xks-ca" pod "492e1026-b99a-4e7b-a65a-a89fbfb6dac2" (UID: "492e1026-b99a-4e7b-a65a-a89fbfb6dac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:57.510619 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.510604    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-kube-api-access-ftwh2" pod "492e1026-b99a-4e7b-a65a-a89fbfb6dac2" (UID: "492e1026-b99a-4e7b-a65a-a89fbfb6dac2"). InnerVolumeSpecName "kube-api-access-ftwh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:57.610661 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.610630    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftwh2\" (UniqueName: \"kubernetes.io/projected/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-kube-api-access-ftwh2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:57.610661 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:57.610660    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492e1026-b99a-4e7b-a65a-a89fbfb6dac2-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.254309 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:58.254282    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"
Oct 31 02:16:58.254504 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:58.254317    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx" event={"ID":"492e1026-b99a-4e7b-a65a-a89fbfb6dac2","Type":"ContainerDied","Data":"9e8d1c5b9bf317952be3c62fba4807bf3f8ced5fef55283a276a1b945146d186"}
Oct 31 02:16:58.254504 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:58.254367    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e8d1c5b9bf317952be3c62fba4807bf3f8ced5fef55283a276a1b945146d186"
Oct 31 02:16:59.255954 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:59.255910    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"]
Oct 31 02:16:59.261910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:59.261881    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bb5c51-bnx7b"]
Oct 31 02:16:59.443408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:59.443354    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5" podUID="3419e3d8-bec8-4eba-9e85-dd542ed634dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:16:59.444292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:16:59.443802    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6" podUID="eb03a89a-bd26-436c-9669-9a289b79dad1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:00.177077 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:00.177018    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3367513-7f0c-4daf-a1d3-3c8893feeab2" path="/var/lib/kubelet/pods/b3367513-7f0c-4daf-a1d3-3c8893feeab2/volumes"
Oct 31 02:17:05.732985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.732947    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"]
Oct 31 02:17:05.733769 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.733749    5293 cpu_manager.go:414] "containerMap: removing container" podUID="492e1026-b99a-4e7b-a65a-a89fbfb6dac2" containerName="revoke-keys"
Oct 31 02:17:05.733864 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.733770    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492e1026-b99a-4e7b-a65a-a89fbfb6dac2" containerName="revoke-keys"
Oct 31 02:17:05.733972 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.733960    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="492e1026-b99a-4e7b-a65a-a89fbfb6dac2" containerName="revoke-keys"
Oct 31 02:17:05.739247 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.739224    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.758744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.758725    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:17:05.759965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.759942    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:17:05.764980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.764951    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"]
Oct 31 02:17:05.869799 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.869768    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb84x\" (UniqueName: \"kubernetes.io/projected/4f65b85e-54ea-4b79-8199-b77eb22adcbb-kube-api-access-xb84x\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.869993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.869818    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.870166 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.870142    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.972005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.971970    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.972220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.972126    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb84x\" (UniqueName: \"kubernetes.io/projected/4f65b85e-54ea-4b79-8199-b77eb22adcbb-kube-api-access-xb84x\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.972220 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.972147    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.972630 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.972609    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.974978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.974948    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:05.987656 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:05.987598    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb84x\" (UniqueName: \"kubernetes.io/projected/4f65b85e-54ea-4b79-8199-b77eb22adcbb-kube-api-access-xb84x\") pod \"maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:06.056051 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:06.056021    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:06.349716 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:06.349692    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"]
Oct 31 02:17:06.351604 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:17:06.351574    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f65b85e_54ea_4b79_8199_b77eb22adcbb.slice/crio-1c3fe95d7c51e7a356d84e615196afcad63684496b966587b842fc066d4686c0 WatchSource:0}: Error finding container 1c3fe95d7c51e7a356d84e615196afcad63684496b966587b842fc066d4686c0: Status 404 returned error can't find the container with id 1c3fe95d7c51e7a356d84e615196afcad63684496b966587b842fc066d4686c0
Oct 31 02:17:06.971515 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:06.971437    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:17:07.367941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:07.367843    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8" event={"ID":"4f65b85e-54ea-4b79-8199-b77eb22adcbb","Type":"ContainerStarted","Data":"f6d60edb932cc85388971f05e38fcf64702f75803d69ce5586f6a0d5e0413ef4"}
Oct 31 02:17:07.367941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:07.367907    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8" event={"ID":"4f65b85e-54ea-4b79-8199-b77eb22adcbb","Type":"ContainerStarted","Data":"1c3fe95d7c51e7a356d84e615196afcad63684496b966587b842fc066d4686c0"}
Oct 31 02:17:07.392370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:07.392323    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8" podStartSLOduration=2.392310093 podStartE2EDuration="2.392310093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:17:07.390204425 +0000 UTC m=+3385.338852799" watchObservedRunningTime="2026-10-31 02:17:07.392310093 +0000 UTC m=+3385.340958521"
Oct 31 02:17:08.225727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:08.225688    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"]
Oct 31 02:17:08.226284 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:08.225956    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy" containerID="cri-o://19e5a8fff6e3717eae00cd740ae2c8fd4d1f2054c5d9bd714b7d1715acaf9f72" gracePeriod=30
Oct 31 02:17:08.441851 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:08.440428    5293 generic.go:356] "Generic (PLEG): container finished" podID="4f65b85e-54ea-4b79-8199-b77eb22adcbb" containerID="f6d60edb932cc85388971f05e38fcf64702f75803d69ce5586f6a0d5e0413ef4" exitCode=0
Oct 31 02:17:08.441851 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:08.440592    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8" event={"ID":"4f65b85e-54ea-4b79-8199-b77eb22adcbb","Type":"ContainerDied","Data":"f6d60edb932cc85388971f05e38fcf64702f75803d69ce5586f6a0d5e0413ef4"}
Oct 31 02:17:09.442863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.442824    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5" podUID="3419e3d8-bec8-4eba-9e85-dd542ed634dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:09.443384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.443094    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6" podUID="eb03a89a-bd26-436c-9669-9a289b79dad1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:09.659958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.659925    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:09.765728 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.765650    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-xks-ca\") pod \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") "
Oct 31 02:17:09.765902 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.765837    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f65b85e-54ea-4b79-8199-b77eb22adcbb-kube-api-access-xb84x\" (UniqueName: \"kubernetes.io/projected/4f65b85e-54ea-4b79-8199-b77eb22adcbb-kube-api-access-xb84x\") pod \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") "
Oct 31 02:17:09.765902 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.765868    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-openshift-ca\") pod \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\" (UID: \"4f65b85e-54ea-4b79-8199-b77eb22adcbb\") "
Oct 31 02:17:09.766346 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.766315    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-openshift-ca" pod "4f65b85e-54ea-4b79-8199-b77eb22adcbb" (UID: "4f65b85e-54ea-4b79-8199-b77eb22adcbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:09.766643 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.766544    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:09.768025 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.768001    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-xks-ca" pod "4f65b85e-54ea-4b79-8199-b77eb22adcbb" (UID: "4f65b85e-54ea-4b79-8199-b77eb22adcbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:09.768286 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.768266    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f65b85e-54ea-4b79-8199-b77eb22adcbb-kube-api-access-xb84x" pod "4f65b85e-54ea-4b79-8199-b77eb22adcbb" (UID: "4f65b85e-54ea-4b79-8199-b77eb22adcbb"). InnerVolumeSpecName "kube-api-access-xb84x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:09.868367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.868337    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb84x\" (UniqueName: \"kubernetes.io/projected/4f65b85e-54ea-4b79-8199-b77eb22adcbb-kube-api-access-xb84x\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:09.868367 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:09.868363    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f65b85e-54ea-4b79-8199-b77eb22adcbb-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:10.458562 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:10.458461    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"
Oct 31 02:17:10.458562 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:10.458491    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8" event={"ID":"4f65b85e-54ea-4b79-8199-b77eb22adcbb","Type":"ContainerDied","Data":"1c3fe95d7c51e7a356d84e615196afcad63684496b966587b842fc066d4686c0"}
Oct 31 02:17:10.458562 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:10.458548    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c3fe95d7c51e7a356d84e615196afcad63684496b966587b842fc066d4686c0"
Oct 31 02:17:11.797296 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:11.797255    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:17:12.564076 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:12.559374    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"]
Oct 31 02:17:12.565372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:12.565324    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d0c6ff-58vvx"]
Oct 31 02:17:13.524112 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.523964    5293 generic.go:356] "Generic (PLEG): container finished" podID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerID="19e5a8fff6e3717eae00cd740ae2c8fd4d1f2054c5d9bd714b7d1715acaf9f72" exitCode=0
Oct 31 02:17:13.524830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.524112    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" event={"ID":"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75","Type":"ContainerDied","Data":"19e5a8fff6e3717eae00cd740ae2c8fd4d1f2054c5d9bd714b7d1715acaf9f72"}
Oct 31 02:17:13.556215 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.556193    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:17:13.576207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.576175    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-credential-socket\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.576451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.576428    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-data\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.576563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.576537    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-credential-socket" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.576678 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.576660    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-token\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-token\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.576827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.576796    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-data" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.576973 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.576934    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-podinfo\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.577198 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.577178    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-socket\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.577360 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.577345    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-envoy\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.577920 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.577900    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-certs\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.578205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.578184    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-kube-api-access-g9nmt\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-kube-api-access-g9nmt\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.578364 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.578347    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istiod-ca-cert\") pod \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\" (UID: \"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75\") "
Oct 31 02:17:13.579563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.577455    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-socket" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.579563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.579298    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-certs" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.590290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.585561    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.590290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.585594    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.590290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.585610    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.590290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.585624    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.590290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.585981    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istiod-ca-cert" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:13.590290 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.590136    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-envoy" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.590840 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.590302    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-token" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:13.590840 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.590339    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-podinfo" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:13.593501 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.593472    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-kube-api-access-g9nmt" pod "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" (UID: "d0ac2bc5-bbbe-4af0-91c6-02bad5837a75"). InnerVolumeSpecName "kube-api-access-g9nmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:13.687950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.687843    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9nmt\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-kube-api-access-g9nmt\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.687950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.687879    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.687950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.687894    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.687950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.687910    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.687950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:13.687924    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:14.175032 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:14.174999    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492e1026-b99a-4e7b-a65a-a89fbfb6dac2" path="/var/lib/kubelet/pods/492e1026-b99a-4e7b-a65a-a89fbfb6dac2/volumes"
Oct 31 02:17:14.536663 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:14.536575    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"
Oct 31 02:17:14.537174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:14.536577    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt" event={"ID":"d0ac2bc5-bbbe-4af0-91c6-02bad5837a75","Type":"ContainerDied","Data":"968c2d7df8b19a3a3630fac5a2985625ae78452c241c0535ddddfc16a64f9882"}
Oct 31 02:17:14.537174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:14.536716    5293 scope.go:122] "RemoveContainer" containerID="19e5a8fff6e3717eae00cd740ae2c8fd4d1f2054c5d9bd714b7d1715acaf9f72"
Oct 31 02:17:14.559452 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:14.559430    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"]
Oct 31 02:17:14.565164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:14.565140    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-x4mpt"]
Oct 31 02:17:15.326995 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.326646    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:17:15.395254 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.395229    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:17:15.412106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.412078    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3419e3d8-bec8-4eba-9e85-dd542ed634dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3419e3d8-bec8-4eba-9e85-dd542ed634dc-plugins-config-volume\") pod \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\" (UID: \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\") "
Oct 31 02:17:15.412264 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.412193    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb03a89a-bd26-436c-9669-9a289b79dad1-kube-api-access-v8b2c\" (UniqueName: \"kubernetes.io/projected/eb03a89a-bd26-436c-9669-9a289b79dad1-kube-api-access-v8b2c\") pod \"eb03a89a-bd26-436c-9669-9a289b79dad1\" (UID: \"eb03a89a-bd26-436c-9669-9a289b79dad1\") "
Oct 31 02:17:15.412327 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.412274    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3419e3d8-bec8-4eba-9e85-dd542ed634dc-kube-api-access-zlrzg\" (UniqueName: \"kubernetes.io/projected/3419e3d8-bec8-4eba-9e85-dd542ed634dc-kube-api-access-zlrzg\") pod \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\" (UID: \"3419e3d8-bec8-4eba-9e85-dd542ed634dc\") "
Oct 31 02:17:15.412394 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.412377    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb03a89a-bd26-436c-9669-9a289b79dad1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb03a89a-bd26-436c-9669-9a289b79dad1-plugins-config-volume\") pod \"eb03a89a-bd26-436c-9669-9a289b79dad1\" (UID: \"eb03a89a-bd26-436c-9669-9a289b79dad1\") "
Oct 31 02:17:15.412479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.412451    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3419e3d8-bec8-4eba-9e85-dd542ed634dc-plugins-config-volume" pod "3419e3d8-bec8-4eba-9e85-dd542ed634dc" (UID: "3419e3d8-bec8-4eba-9e85-dd542ed634dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:15.412854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.412825    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb03a89a-bd26-436c-9669-9a289b79dad1-plugins-config-volume" pod "eb03a89a-bd26-436c-9669-9a289b79dad1" (UID: "eb03a89a-bd26-436c-9669-9a289b79dad1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:15.413027 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.413005    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3419e3d8-bec8-4eba-9e85-dd542ed634dc-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:15.413027 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.413021    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb03a89a-bd26-436c-9669-9a289b79dad1-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:15.416538 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.415891    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb03a89a-bd26-436c-9669-9a289b79dad1-kube-api-access-v8b2c" pod "eb03a89a-bd26-436c-9669-9a289b79dad1" (UID: "eb03a89a-bd26-436c-9669-9a289b79dad1"). InnerVolumeSpecName "kube-api-access-v8b2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:15.416538 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.415941    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3419e3d8-bec8-4eba-9e85-dd542ed634dc-kube-api-access-zlrzg" pod "3419e3d8-bec8-4eba-9e85-dd542ed634dc" (UID: "3419e3d8-bec8-4eba-9e85-dd542ed634dc"). InnerVolumeSpecName "kube-api-access-zlrzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:15.514457 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.514418    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8b2c\" (UniqueName: \"kubernetes.io/projected/eb03a89a-bd26-436c-9669-9a289b79dad1-kube-api-access-v8b2c\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:15.514457 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.514445    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrzg\" (UniqueName: \"kubernetes.io/projected/3419e3d8-bec8-4eba-9e85-dd542ed634dc-kube-api-access-zlrzg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:15.548820 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.548790    5293 generic.go:356] "Generic (PLEG): container finished" podID="eb03a89a-bd26-436c-9669-9a289b79dad1" containerID="8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d" exitCode=0
Oct 31 02:17:15.549299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.548902    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"
Oct 31 02:17:15.549299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.548893    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6" event={"ID":"eb03a89a-bd26-436c-9669-9a289b79dad1","Type":"ContainerDied","Data":"8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d"}
Oct 31 02:17:15.549299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.549015    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6" event={"ID":"eb03a89a-bd26-436c-9669-9a289b79dad1","Type":"ContainerDied","Data":"0a94d32d0e3ccf05db84e9214904571ca92448d3a93ee7866d3acbf10645cc2e"}
Oct 31 02:17:15.549299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.549056    5293 scope.go:122] "RemoveContainer" containerID="8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d"
Oct 31 02:17:15.550498 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.550476    5293 generic.go:356] "Generic (PLEG): container finished" podID="3419e3d8-bec8-4eba-9e85-dd542ed634dc" containerID="5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11" exitCode=0
Oct 31 02:17:15.550602 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.550523    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"
Oct 31 02:17:15.550602 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.550532    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5" event={"ID":"3419e3d8-bec8-4eba-9e85-dd542ed634dc","Type":"ContainerDied","Data":"5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11"}
Oct 31 02:17:15.550602 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.550580    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5" event={"ID":"3419e3d8-bec8-4eba-9e85-dd542ed634dc","Type":"ContainerDied","Data":"ce9447b900a7371b9f219c4dcfd44e5dece04bd13cbb98ffd0968bbb3757c7b4"}
Oct 31 02:17:15.560313 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.560296    5293 scope.go:122] "RemoveContainer" containerID="8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d"
Oct 31 02:17:15.560606 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:15.560586    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d\": container with ID starting with 8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d not found: ID does not exist" containerID="8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d"
Oct 31 02:17:15.560693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.560613    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d"} err="failed to get container status \"8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d\": rpc error: code = NotFound desc = could not find container \"8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d\": container with ID starting with 8cf41cf8376f159371483cfd135d2701949b7b163a516002173fdaa8f307455d not found: ID does not exist"
Oct 31 02:17:15.560693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.560631    5293 scope.go:122] "RemoveContainer" containerID="5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11"
Oct 31 02:17:15.570579 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.570558    5293 scope.go:122] "RemoveContainer" containerID="5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11"
Oct 31 02:17:15.570856 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:15.570836    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11\": container with ID starting with 5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11 not found: ID does not exist" containerID="5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11"
Oct 31 02:17:15.570918 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.570866    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11"} err="failed to get container status \"5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11\": rpc error: code = NotFound desc = could not find container \"5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11\": container with ID starting with 5f3c7c1ef02a551cf89e577b6cd3cecbfd0f53998db2d9c55771fb186e607e11 not found: ID does not exist"
Oct 31 02:17:15.578982 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.578928    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"]
Oct 31 02:17:15.582596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.582575    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bb5c51-547cf58c9-m4fp5"]
Oct 31 02:17:15.593029 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.593007    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"]
Oct 31 02:17:15.601267 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.601244    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bb5c51-7b67ddc79-q6nc6"]
Oct 31 02:17:15.857421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.857341    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"]
Oct 31 02:17:15.872096 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:15.872066    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"]
Oct 31 02:17:16.213066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:16.213005    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3419e3d8-bec8-4eba-9e85-dd542ed634dc" path="/var/lib/kubelet/pods/3419e3d8-bec8-4eba-9e85-dd542ed634dc/volumes"
Oct 31 02:17:16.222065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:16.221808    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" path="/var/lib/kubelet/pods/d0ac2bc5-bbbe-4af0-91c6-02bad5837a75/volumes"
Oct 31 02:17:16.229064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:16.222497    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb03a89a-bd26-436c-9669-9a289b79dad1" path="/var/lib/kubelet/pods/eb03a89a-bd26-436c-9669-9a289b79dad1/volumes"
Oct 31 02:17:18.311927 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:18.311880    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"]
Oct 31 02:17:18.315314 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:18.315284    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-28e4f3-mqmk8"]
Oct 31 02:17:18.908188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:18.908155    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"]
Oct 31 02:17:20.175522 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:20.175489    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f65b85e-54ea-4b79-8199-b77eb22adcbb" path="/var/lib/kubelet/pods/4f65b85e-54ea-4b79-8199-b77eb22adcbb/volumes"
Oct 31 02:17:20.659394 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:20.659362    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"]
Oct 31 02:17:20.659627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:20.659596    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy" containerID="cri-o://4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6" gracePeriod=30
Oct 31 02:17:20.861070 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:20.860123    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h" podUID="0a86ec52-6276-48bd-9aa3-bf5ad261165e" containerName="payload-processing" containerID="cri-o://868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61" gracePeriod=25
Oct 31 02:17:20.877328 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:20.877120    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s" podUID="af726e94-0607-4529-a9da-da1d698ff6ca" containerName="payload-pre-processing" containerID="cri-o://89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445" gracePeriod=25
Oct 31 02:17:22.686887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:22.686847    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"]
Oct 31 02:17:22.687317 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:22.687125    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8" podUID="18654831-3ecb-46ae-bd66-59c4725008fe" containerName="maas-api" containerID="cri-o://1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80" gracePeriod=30
Oct 31 02:17:23.215120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.215091    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:17:23.335664 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.335620    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18654831-3ecb-46ae-bd66-59c4725008fe-kube-api-access-ztvpm\" (UniqueName: \"kubernetes.io/projected/18654831-3ecb-46ae-bd66-59c4725008fe-kube-api-access-ztvpm\") pod \"18654831-3ecb-46ae-bd66-59c4725008fe\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") "
Oct 31 02:17:23.335870 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.335741    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18654831-3ecb-46ae-bd66-59c4725008fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18654831-3ecb-46ae-bd66-59c4725008fe-openshift-service-ca\") pod \"18654831-3ecb-46ae-bd66-59c4725008fe\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") "
Oct 31 02:17:23.335946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.335871    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-metrics-certs\") pod \"18654831-3ecb-46ae-bd66-59c4725008fe\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") "
Oct 31 02:17:23.335946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.335938    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-maas-api-tls\") pod \"18654831-3ecb-46ae-bd66-59c4725008fe\" (UID: \"18654831-3ecb-46ae-bd66-59c4725008fe\") "
Oct 31 02:17:23.336315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.336284    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18654831-3ecb-46ae-bd66-59c4725008fe-openshift-service-ca" pod "18654831-3ecb-46ae-bd66-59c4725008fe" (UID: "18654831-3ecb-46ae-bd66-59c4725008fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:23.336594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.336574    5293 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18654831-3ecb-46ae-bd66-59c4725008fe-openshift-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:23.338467 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.338436    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-maas-api-tls" pod "18654831-3ecb-46ae-bd66-59c4725008fe" (UID: "18654831-3ecb-46ae-bd66-59c4725008fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:23.338700 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.338677    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-metrics-certs" pod "18654831-3ecb-46ae-bd66-59c4725008fe" (UID: "18654831-3ecb-46ae-bd66-59c4725008fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:23.338798 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.338710    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18654831-3ecb-46ae-bd66-59c4725008fe-kube-api-access-ztvpm" pod "18654831-3ecb-46ae-bd66-59c4725008fe" (UID: "18654831-3ecb-46ae-bd66-59c4725008fe"). InnerVolumeSpecName "kube-api-access-ztvpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:23.438389 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.438299    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvpm\" (UniqueName: \"kubernetes.io/projected/18654831-3ecb-46ae-bd66-59c4725008fe-kube-api-access-ztvpm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:23.438389 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.438333    5293 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-metrics-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:23.438389 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.438344    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18654831-3ecb-46ae-bd66-59c4725008fe-maas-api-tls\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:23.666435 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.666397    5293 generic.go:356] "Generic (PLEG): container finished" podID="18654831-3ecb-46ae-bd66-59c4725008fe" containerID="1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80" exitCode=0
Oct 31 02:17:23.666639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.666453    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"
Oct 31 02:17:23.666639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.666486    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8" event={"ID":"18654831-3ecb-46ae-bd66-59c4725008fe","Type":"ContainerDied","Data":"1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80"}
Oct 31 02:17:23.666639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.666548    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8" event={"ID":"18654831-3ecb-46ae-bd66-59c4725008fe","Type":"ContainerDied","Data":"44cdb7e46bc00e82d4ccb2f9ee61d01254447433aaa5272560ded17bf0f4a7aa"}
Oct 31 02:17:23.666639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.666577    5293 scope.go:122] "RemoveContainer" containerID="1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80"
Oct 31 02:17:23.677034 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.677015    5293 scope.go:122] "RemoveContainer" containerID="1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80"
Oct 31 02:17:23.677357 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:23.677333    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80\": container with ID starting with 1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80 not found: ID does not exist" containerID="1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80"
Oct 31 02:17:23.677484 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.677365    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80"} err="failed to get container status \"1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80\": rpc error: code = NotFound desc = could not find container \"1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80\": container with ID starting with 1745ce6c4ca6d1ee417e85e9f39699090f8225e387a493bb771a4c4323d39d80 not found: ID does not exist"
Oct 31 02:17:23.688733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.688678    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"]
Oct 31 02:17:23.693783 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.693760    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8"]
Oct 31 02:17:23.908535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.908497    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:17:23.908775 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:23.908747    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct" podUID="1f67d3a9-5038-4167-8a95-4d23fcd86719" containerName="payload-processing" containerID="cri-o://f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723" gracePeriod=25
Oct 31 02:17:24.175190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:24.175155    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18654831-3ecb-46ae-bd66-59c4725008fe" path="/var/lib/kubelet/pods/18654831-3ecb-46ae-bd66-59c4725008fe/volumes"
Oct 31 02:17:25.986526 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:25.986497    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:17:26.097300 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097261    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-data\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.097477 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097365    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-kube-api-access-rtc7s\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-kube-api-access-rtc7s\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.097477 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097415    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-credential-socket\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.097565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097541    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-socket\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.097692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097554    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-data" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.097692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097618    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-istio-token\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-istio-token\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.097692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097654    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eaac94e-52de-410f-a032-7615ba6c9992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaac94e-52de-410f-a032-7615ba6c9992-istio-podinfo\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.097876 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097697    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-credential-socket" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.097876 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097781    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-certs\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.097876 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097819    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-envoy\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.097876 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097864    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eaac94e-52de-410f-a032-7615ba6c9992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaac94e-52de-410f-a032-7615ba6c9992-istiod-ca-cert\") pod \"1eaac94e-52de-410f-a032-7615ba6c9992\" (UID: \"1eaac94e-52de-410f-a032-7615ba6c9992\") "
Oct 31 02:17:26.098138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.097897    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-socket" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.098138 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.098115    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-certs" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.098755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.098679    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.098755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.098711    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.098755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.098697    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eaac94e-52de-410f-a032-7615ba6c9992-istiod-ca-cert" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:26.098755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.098727    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.098755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.098742    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.100103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.100075    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eaac94e-52de-410f-a032-7615ba6c9992-istio-podinfo" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:26.100201 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.100109    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-istio-token" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:26.100319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.100305    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-kube-api-access-rtc7s" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "kube-api-access-rtc7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:26.100365 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.100328    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-envoy" pod "1eaac94e-52de-410f-a032-7615ba6c9992" (UID: "1eaac94e-52de-410f-a032-7615ba6c9992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.140289 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.136090    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-24qp8" podUID="18654831-3ecb-46ae-bd66-59c4725008fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.58:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:17:26.186324 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.186292    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"]
Oct 31 02:17:26.186551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.186527    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy" containerID="cri-o://5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a" gracePeriod=30
Oct 31 02:17:26.199904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.199844    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.199904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.199873    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaac94e-52de-410f-a032-7615ba6c9992-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.199904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.199886    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaac94e-52de-410f-a032-7615ba6c9992-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.199904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.199899    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaac94e-52de-410f-a032-7615ba6c9992-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.200136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.199913    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtc7s\" (UniqueName: \"kubernetes.io/projected/1eaac94e-52de-410f-a032-7615ba6c9992-kube-api-access-rtc7s\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.724628 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.724594    5293 generic.go:356] "Generic (PLEG): container finished" podID="1eaac94e-52de-410f-a032-7615ba6c9992" containerID="4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6" exitCode=0
Oct 31 02:17:26.724830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.724661    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" event={"ID":"1eaac94e-52de-410f-a032-7615ba6c9992","Type":"ContainerDied","Data":"4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6"}
Oct 31 02:17:26.724830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.724683    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"
Oct 31 02:17:26.724830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.724707    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm" event={"ID":"1eaac94e-52de-410f-a032-7615ba6c9992","Type":"ContainerDied","Data":"47b9d1b280a322409f020b4c086ca1cc1931dbb2d6f85f7669130a465ba65f27"}
Oct 31 02:17:26.724830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.724728    5293 scope.go:122] "RemoveContainer" containerID="4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6"
Oct 31 02:17:26.736424 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.736400    5293 scope.go:122] "RemoveContainer" containerID="1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad"
Oct 31 02:17:26.747273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.747248    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"]
Oct 31 02:17:26.747929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.747912    5293 scope.go:122] "RemoveContainer" containerID="4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6"
Oct 31 02:17:26.748215 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:26.748189    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6\": container with ID starting with 4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6 not found: ID does not exist" containerID="4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6"
Oct 31 02:17:26.748315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.748229    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6"} err="failed to get container status \"4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6\": rpc error: code = NotFound desc = could not find container \"4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6\": container with ID starting with 4ca6c603f0fa230c52759bc4cb6f28186e52093f2f7e8ac845fa4b2d0dbec7a6 not found: ID does not exist"
Oct 31 02:17:26.748315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.748253    5293 scope.go:122] "RemoveContainer" containerID="1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad"
Oct 31 02:17:26.748545 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:26.748526    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad\": container with ID starting with 1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad not found: ID does not exist" containerID="1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad"
Oct 31 02:17:26.748609 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.748564    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad"} err="failed to get container status \"1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad\": rpc error: code = NotFound desc = could not find container \"1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad\": container with ID starting with 1cdf74e2e874032f21dc303fb512da201571d59b921fb93275f446b6624769ad not found: ID does not exist"
Oct 31 02:17:26.752792 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.752767    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-m95mm"]
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.891966    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"]
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894419    5293 cpu_manager.go:414] "containerMap: removing container" podUID="eb03a89a-bd26-436c-9669-9a289b79dad1" containerName="payload-pre-processing"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894438    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb03a89a-bd26-436c-9669-9a289b79dad1" containerName="payload-pre-processing"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894456    5293 cpu_manager.go:414] "containerMap: removing container" podUID="18654831-3ecb-46ae-bd66-59c4725008fe" containerName="maas-api"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894464    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18654831-3ecb-46ae-bd66-59c4725008fe" containerName="maas-api"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894480    5293 cpu_manager.go:414] "containerMap: removing container" podUID="4f65b85e-54ea-4b79-8199-b77eb22adcbb" containerName="revoke-keys"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894487    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f65b85e-54ea-4b79-8199-b77eb22adcbb" containerName="revoke-keys"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894508    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894515    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894533    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3419e3d8-bec8-4eba-9e85-dd542ed634dc" containerName="payload-processing"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894540    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3419e3d8-bec8-4eba-9e85-dd542ed634dc" containerName="payload-processing"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894572    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894581    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894602    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894609    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894724    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894739    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb03a89a-bd26-436c-9669-9a289b79dad1" containerName="payload-pre-processing"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894749    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="18654831-3ecb-46ae-bd66-59c4725008fe" containerName="maas-api"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894759    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f65b85e-54ea-4b79-8199-b77eb22adcbb" containerName="revoke-keys"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894769    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ac2bc5-bbbe-4af0-91c6-02bad5837a75" containerName="istio-proxy"
Oct 31 02:17:26.895235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.894777    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3419e3d8-bec8-4eba-9e85-dd542ed634dc" containerName="payload-processing"
Oct 31 02:17:26.902448 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.902423    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"]
Oct 31 02:17:26.902623 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.902534    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:26.907877 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.907141    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:17:26.907877 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:26.907239    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:17:27.025600 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.025514    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.025974 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.025701    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sbd\" (UniqueName: \"kubernetes.io/projected/a780f315-2f01-4336-958d-c0b59d03e9f3-kube-api-access-l5sbd\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.025974 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.025750    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.127129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.127078    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.127336 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.127228    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sbd\" (UniqueName: \"kubernetes.io/projected/a780f315-2f01-4336-958d-c0b59d03e9f3-kube-api-access-l5sbd\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.127336 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.127283    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.127881 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.127861    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.129927 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.129904    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.140597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.140567    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sbd\" (UniqueName: \"kubernetes.io/projected/a780f315-2f01-4336-958d-c0b59d03e9f3-kube-api-access-l5sbd\") pod \"maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.224176 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.224143    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:27.520070 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:17:27.520004    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda780f315_2f01_4336_958d_c0b59d03e9f3.slice/crio-cf128d34e6c396b893b53613dffef28531257df83a3e920b4d6e71397c70cdcc WatchSource:0}: Error finding container cf128d34e6c396b893b53613dffef28531257df83a3e920b4d6e71397c70cdcc: Status 404 returned error can't find the container with id cf128d34e6c396b893b53613dffef28531257df83a3e920b4d6e71397c70cdcc
Oct 31 02:17:27.520225 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.520127    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"]
Oct 31 02:17:27.734638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:27.734601    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8" event={"ID":"a780f315-2f01-4336-958d-c0b59d03e9f3","Type":"ContainerStarted","Data":"cf128d34e6c396b893b53613dffef28531257df83a3e920b4d6e71397c70cdcc"}
Oct 31 02:17:28.195075 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:28.189587    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" path="/var/lib/kubelet/pods/1eaac94e-52de-410f-a032-7615ba6c9992/volumes"
Oct 31 02:17:28.743974 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:28.743945    5293 generic.go:356] "Generic (PLEG): container finished" podID="a780f315-2f01-4336-958d-c0b59d03e9f3" containerID="75c8cb6e877da21f4948764199b96154289f263143ee911b0a82dd26ed5724e0" exitCode=0
Oct 31 02:17:28.744183 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:28.744035    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8" event={"ID":"a780f315-2f01-4336-958d-c0b59d03e9f3","Type":"ContainerDied","Data":"75c8cb6e877da21f4948764199b96154289f263143ee911b0a82dd26ed5724e0"}
Oct 31 02:17:29.941614 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:29.941587    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:30.079364 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.079273    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-openshift-ca\") pod \"a780f315-2f01-4336-958d-c0b59d03e9f3\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") "
Oct 31 02:17:30.079540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.079408    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a780f315-2f01-4336-958d-c0b59d03e9f3-kube-api-access-l5sbd\" (UniqueName: \"kubernetes.io/projected/a780f315-2f01-4336-958d-c0b59d03e9f3-kube-api-access-l5sbd\") pod \"a780f315-2f01-4336-958d-c0b59d03e9f3\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") "
Oct 31 02:17:30.079540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.079443    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-xks-ca\") pod \"a780f315-2f01-4336-958d-c0b59d03e9f3\" (UID: \"a780f315-2f01-4336-958d-c0b59d03e9f3\") "
Oct 31 02:17:30.079692 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.079661    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-openshift-ca" pod "a780f315-2f01-4336-958d-c0b59d03e9f3" (UID: "a780f315-2f01-4336-958d-c0b59d03e9f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:30.079956 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.079939    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:30.081659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.081637    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a780f315-2f01-4336-958d-c0b59d03e9f3-kube-api-access-l5sbd" pod "a780f315-2f01-4336-958d-c0b59d03e9f3" (UID: "a780f315-2f01-4336-958d-c0b59d03e9f3"). InnerVolumeSpecName "kube-api-access-l5sbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:30.081756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.081733    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-xks-ca" pod "a780f315-2f01-4336-958d-c0b59d03e9f3" (UID: "a780f315-2f01-4336-958d-c0b59d03e9f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:30.181618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.181585    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5sbd\" (UniqueName: \"kubernetes.io/projected/a780f315-2f01-4336-958d-c0b59d03e9f3-kube-api-access-l5sbd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:30.181618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.181614    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780f315-2f01-4336-958d-c0b59d03e9f3-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:30.285574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.285532    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h" podUID="0a86ec52-6276-48bd-9aa3-bf5ad261165e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:30.286291 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.286226    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s" podUID="af726e94-0607-4529-a9da-da1d698ff6ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:30.615082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.602335    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"]
Oct 31 02:17:30.615082 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.602610    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy" containerID="cri-o://a580021edf03119adaf7527a7876de0e89b6762c460417a138db5374150c8793" gracePeriod=30
Oct 31 02:17:30.762066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.714009    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct" podUID="1f67d3a9-5038-4167-8a95-4d23fcd86719" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:30.831065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.823102    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8" event={"ID":"a780f315-2f01-4336-958d-c0b59d03e9f3","Type":"ContainerDied","Data":"cf128d34e6c396b893b53613dffef28531257df83a3e920b4d6e71397c70cdcc"}
Oct 31 02:17:30.831065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.823169    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf128d34e6c396b893b53613dffef28531257df83a3e920b4d6e71397c70cdcc"
Oct 31 02:17:30.831065 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:30.823241    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"
Oct 31 02:17:31.370270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.369165    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"]
Oct 31 02:17:31.370270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.370094    5293 cpu_manager.go:414] "containerMap: removing container" podUID="a780f315-2f01-4336-958d-c0b59d03e9f3" containerName="revoke-keys"
Oct 31 02:17:31.370270 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.370116    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a780f315-2f01-4336-958d-c0b59d03e9f3" containerName="revoke-keys"
Oct 31 02:17:31.386556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.370583    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eaac94e-52de-410f-a032-7615ba6c9992" containerName="istio-proxy"
Oct 31 02:17:31.386556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.370615    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="a780f315-2f01-4336-958d-c0b59d03e9f3" containerName="revoke-keys"
Oct 31 02:17:31.386556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.374695    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.386556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.378525    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:17:31.386556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.379986    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:17:31.388321 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.388145    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"]
Oct 31 02:17:31.518578 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.518548    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.518738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.518705    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.518805 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.518773    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwktm\" (UniqueName: \"kubernetes.io/projected/9634500c-51f5-4687-99f6-a446c6bae152-kube-api-access-kwktm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.537949 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.537922    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:17:31.630908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.628187    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-socket\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.657064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.652648    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-socket" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:31.661617 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.658430    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-credential-socket" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:31.664064 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.663106    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-credential-socket\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.690669    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-kube-api-access-fbjlk\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-kube-api-access-fbjlk\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.690751    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-envoy\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.691326    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e177db-8776-47df-96ad-404ec445b4b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e177db-8776-47df-96ad-404ec445b4b6-istiod-ca-cert\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.692287    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e177db-8776-47df-96ad-404ec445b4b6-istiod-ca-cert" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.692434    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-data\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.692739    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-data" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.692467    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-istio-token\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-istio-token\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.693115    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8e177db-8776-47df-96ad-404ec445b4b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e177db-8776-47df-96ad-404ec445b4b6-istio-podinfo\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.703066 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.699092    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-certs\") pod \"e8e177db-8776-47df-96ad-404ec445b4b6\" (UID: \"e8e177db-8776-47df-96ad-404ec445b4b6\") "
Oct 31 02:17:31.705655 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.705578    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-certs" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:31.708890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.706232    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.708890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.706379    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.708890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.706786    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwktm\" (UniqueName: \"kubernetes.io/projected/9634500c-51f5-4687-99f6-a446c6bae152-kube-api-access-kwktm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.708890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.707073    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e177db-8776-47df-96ad-404ec445b4b6-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.708890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.708334    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.708890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.708355    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.708890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.708371    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.708890 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.708386    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.711031 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.710994    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.751806 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.751742    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8e177db-8776-47df-96ad-404ec445b4b6-istio-podinfo" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:31.757719 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.757629    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.758349 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.758272    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-istio-token" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:31.758727 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.758631    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-envoy" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:31.759014 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.758973    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-kube-api-access-fbjlk" pod "e8e177db-8776-47df-96ad-404ec445b4b6" (UID: "e8e177db-8776-47df-96ad-404ec445b4b6"). InnerVolumeSpecName "kube-api-access-fbjlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:31.765215 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.765134    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwktm\" (UniqueName: \"kubernetes.io/projected/9634500c-51f5-4687-99f6-a446c6bae152-kube-api-access-kwktm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:31.810118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.810003    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjlk\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-kube-api-access-fbjlk\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.810118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.810059    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e177db-8776-47df-96ad-404ec445b4b6-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.810118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.810077    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e177db-8776-47df-96ad-404ec445b4b6-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.810118 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.810092    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e177db-8776-47df-96ad-404ec445b4b6-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:31.833739 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.833703    5293 generic.go:356] "Generic (PLEG): container finished" podID="e8e177db-8776-47df-96ad-404ec445b4b6" containerID="5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a" exitCode=0
Oct 31 02:17:31.833903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.833759    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"
Oct 31 02:17:31.833903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.833794    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" event={"ID":"e8e177db-8776-47df-96ad-404ec445b4b6","Type":"ContainerDied","Data":"5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a"}
Oct 31 02:17:31.833903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.833856    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b" event={"ID":"e8e177db-8776-47df-96ad-404ec445b4b6","Type":"ContainerDied","Data":"c5a68689af00f565544a252cf58972e0cb26baa17c18a12f250ec21a2daaac44"}
Oct 31 02:17:31.833903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.833887    5293 scope.go:122] "RemoveContainer" containerID="5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a"
Oct 31 02:17:31.850685 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.850654    5293 scope.go:122] "RemoveContainer" containerID="5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99"
Oct 31 02:17:31.861635 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.861607    5293 scope.go:122] "RemoveContainer" containerID="5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a"
Oct 31 02:17:31.861948 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:31.861927    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a\": container with ID starting with 5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a not found: ID does not exist" containerID="5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a"
Oct 31 02:17:31.862023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.861958    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a"} err="failed to get container status \"5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a\": rpc error: code = NotFound desc = could not find container \"5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a\": container with ID starting with 5b26b9f5b4c1ac15defb288db75418fdd893f1be2f3b999a166e54c2dd0cc58a not found: ID does not exist"
Oct 31 02:17:31.862023 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.861977    5293 scope.go:122] "RemoveContainer" containerID="5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99"
Oct 31 02:17:31.862302 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:31.862277    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99\": container with ID starting with 5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99 not found: ID does not exist" containerID="5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99"
Oct 31 02:17:31.862346 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.862312    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99"} err="failed to get container status \"5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99\": rpc error: code = NotFound desc = could not find container \"5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99\": container with ID starting with 5d1799ba47a6e3f641cb0b0594010cdadfec80008b450ef28c6b9163cca93d99 not found: ID does not exist"
Oct 31 02:17:31.864681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.864660    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"]
Oct 31 02:17:31.870534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:31.870512    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-28e4f3-openshift-default-5fd84fdcbf-mp22b"]
Oct 31 02:17:31.887931 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:31.887905    5293 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e177db_8776_47df_96ad_404ec445b4b6.slice/crio-c5a68689af00f565544a252cf58972e0cb26baa17c18a12f250ec21a2daaac44\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e177db_8776_47df_96ad_404ec445b4b6.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 02:17:32.028121 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:32.027867    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:32.178987 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:32.178950    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" path="/var/lib/kubelet/pods/e8e177db-8776-47df-96ad-404ec445b4b6/volumes"
Oct 31 02:17:32.370157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:32.370130    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"]
Oct 31 02:17:32.371629 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:17:32.371586    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9634500c_51f5_4687_99f6_a446c6bae152.slice/crio-e763e406ab480328de4ab7d7351c1f6520da6d048fd79cc6ea47e115b05c3966 WatchSource:0}: Error finding container e763e406ab480328de4ab7d7351c1f6520da6d048fd79cc6ea47e115b05c3966: Status 404 returned error can't find the container with id e763e406ab480328de4ab7d7351c1f6520da6d048fd79cc6ea47e115b05c3966
Oct 31 02:17:32.855543 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:32.855499    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq" event={"ID":"9634500c-51f5-4687-99f6-a446c6bae152","Type":"ContainerStarted","Data":"0ff57e23d33bd38d1dab3abda7cfaa068884f5e3beae087fbae37babe6a7c63f"}
Oct 31 02:17:32.855723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:32.855596    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq" event={"ID":"9634500c-51f5-4687-99f6-a446c6bae152","Type":"ContainerStarted","Data":"e763e406ab480328de4ab7d7351c1f6520da6d048fd79cc6ea47e115b05c3966"}
Oct 31 02:17:32.878371 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:32.878327    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq" podStartSLOduration=1.87831036 podStartE2EDuration="1.87831036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:17:32.874518542 +0000 UTC m=+3410.823166934" watchObservedRunningTime="2026-10-31 02:17:32.87831036 +0000 UTC m=+3410.826958751"
Oct 31 02:17:33.864928 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:33.864841    5293 generic.go:356] "Generic (PLEG): container finished" podID="9634500c-51f5-4687-99f6-a446c6bae152" containerID="0ff57e23d33bd38d1dab3abda7cfaa068884f5e3beae087fbae37babe6a7c63f" exitCode=0
Oct 31 02:17:33.865352 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:33.864937    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq" event={"ID":"9634500c-51f5-4687-99f6-a446c6bae152","Type":"ContainerDied","Data":"0ff57e23d33bd38d1dab3abda7cfaa068884f5e3beae087fbae37babe6a7c63f"}
Oct 31 02:17:35.070062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.070021    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:35.168090 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.168056    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9634500c-51f5-4687-99f6-a446c6bae152-kube-api-access-kwktm\" (UniqueName: \"kubernetes.io/projected/9634500c-51f5-4687-99f6-a446c6bae152-kube-api-access-kwktm\") pod \"9634500c-51f5-4687-99f6-a446c6bae152\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") "
Oct 31 02:17:35.168274 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.168194    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-openshift-ca\") pod \"9634500c-51f5-4687-99f6-a446c6bae152\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") "
Oct 31 02:17:35.168423 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.168405    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-xks-ca\") pod \"9634500c-51f5-4687-99f6-a446c6bae152\" (UID: \"9634500c-51f5-4687-99f6-a446c6bae152\") "
Oct 31 02:17:35.168644 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.168618    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-openshift-ca" pod "9634500c-51f5-4687-99f6-a446c6bae152" (UID: "9634500c-51f5-4687-99f6-a446c6bae152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:35.169002 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.168974    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:35.170604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.170577    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9634500c-51f5-4687-99f6-a446c6bae152-kube-api-access-kwktm" pod "9634500c-51f5-4687-99f6-a446c6bae152" (UID: "9634500c-51f5-4687-99f6-a446c6bae152"). InnerVolumeSpecName "kube-api-access-kwktm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:35.170698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.170648    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-xks-ca" pod "9634500c-51f5-4687-99f6-a446c6bae152" (UID: "9634500c-51f5-4687-99f6-a446c6bae152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:35.270233 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.270197    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwktm\" (UniqueName: \"kubernetes.io/projected/9634500c-51f5-4687-99f6-a446c6bae152-kube-api-access-kwktm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:35.270233 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.270228    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9634500c-51f5-4687-99f6-a446c6bae152-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:35.886590 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.886557    5293 generic.go:356] "Generic (PLEG): container finished" podID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerID="a580021edf03119adaf7527a7876de0e89b6762c460417a138db5374150c8793" exitCode=0
Oct 31 02:17:35.886759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.886626    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" event={"ID":"162bf7e4-3a15-4b30-a9c6-f816a6327a12","Type":"ContainerDied","Data":"a580021edf03119adaf7527a7876de0e89b6762c460417a138db5374150c8793"}
Oct 31 02:17:35.886759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.886684    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58" event={"ID":"162bf7e4-3a15-4b30-a9c6-f816a6327a12","Type":"ContainerDied","Data":"33e817ab969c82742480fc10e553f56220b65d7ebc59dbf37fdd7fbd59392b54"}
Oct 31 02:17:35.886759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.886706    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33e817ab969c82742480fc10e553f56220b65d7ebc59dbf37fdd7fbd59392b54"
Oct 31 02:17:35.888078 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.888025    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:17:35.888191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.888174    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq" event={"ID":"9634500c-51f5-4687-99f6-a446c6bae152","Type":"ContainerDied","Data":"e763e406ab480328de4ab7d7351c1f6520da6d048fd79cc6ea47e115b05c3966"}
Oct 31 02:17:35.888234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.888216    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e763e406ab480328de4ab7d7351c1f6520da6d048fd79cc6ea47e115b05c3966"
Oct 31 02:17:35.888288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.888275    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"
Oct 31 02:17:35.983384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983355    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-data\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.983558 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983394    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-envoy\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.983558 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983500    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-kube-api-access-h89nv\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-kube-api-access-h89nv\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.983657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983559    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-podinfo\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.983657 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983640    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-certs\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.983738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983653    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-data" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:35.983788 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983730    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-credential-socket\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.983848 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983808    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-token\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-token\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.983848 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983842    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istiod-ca-cert\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.983975 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.983907    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-socket\") pod \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\" (UID: \"162bf7e4-3a15-4b30-a9c6-f816a6327a12\") "
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984315    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-certs" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984342    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-credential-socket" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984677    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-socket" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984681    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istiod-ca-cert" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984914    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984935    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984951    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984967    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:35.985012 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.984981    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:35.987487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.987462    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-token" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:35.987611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.987501    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-kube-api-access-h89nv" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "kube-api-access-h89nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:35.987611 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.987547    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-podinfo" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:35.988420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:35.988390    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-envoy" pod "162bf7e4-3a15-4b30-a9c6-f816a6327a12" (UID: "162bf7e4-3a15-4b30-a9c6-f816a6327a12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.086307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:36.086274    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h89nv\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-kube-api-access-h89nv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.086307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:36.086303    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.086307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:36.086314    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.086771 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:36.086322    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162bf7e4-3a15-4b30-a9c6-f816a6327a12-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.896437 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:36.896407    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"
Oct 31 02:17:36.917580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:36.917552    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"]
Oct 31 02:17:36.922150 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:36.922125    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-tgc58"]
Oct 31 02:17:37.513684 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:37.513596    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"]
Oct 31 02:17:37.514999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:37.514931    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq" podUID="91cf765a-5a4f-4f39-828e-0aff2dcb96dc" containerName="maas-api" containerID="cri-o://3f9887dc15addf51c66641c64c8d802d6ef9febe05b3645c2d3a5ce0bf5a1e2b" gracePeriod=30
Oct 31 02:17:37.923102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:37.909751    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"]
Oct 31 02:17:37.923102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:37.914994    5293 generic.go:356] "Generic (PLEG): container finished" podID="91cf765a-5a4f-4f39-828e-0aff2dcb96dc" containerID="3f9887dc15addf51c66641c64c8d802d6ef9febe05b3645c2d3a5ce0bf5a1e2b" exitCode=0
Oct 31 02:17:37.923102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:37.915099    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq" event={"ID":"91cf765a-5a4f-4f39-828e-0aff2dcb96dc","Type":"ContainerDied","Data":"3f9887dc15addf51c66641c64c8d802d6ef9febe05b3645c2d3a5ce0bf5a1e2b"}
Oct 31 02:17:37.923102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:37.920527    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c088fd-5ctv8"]
Oct 31 02:17:37.962224 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:37.962199    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:17:38.088469 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.088441    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-maas-api-tls\") pod \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") "
Oct 31 02:17:38.088641 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.088487    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-kube-api-access-sx5vk\" (UniqueName: \"kubernetes.io/projected/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-kube-api-access-sx5vk\") pod \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") "
Oct 31 02:17:38.088641 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.088629    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-metrics-certs\") pod \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") "
Oct 31 02:17:38.088747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.088680    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-openshift-service-ca\") pod \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\" (UID: \"91cf765a-5a4f-4f39-828e-0aff2dcb96dc\") "
Oct 31 02:17:38.089116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.089088    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-openshift-service-ca" pod "91cf765a-5a4f-4f39-828e-0aff2dcb96dc" (UID: "91cf765a-5a4f-4f39-828e-0aff2dcb96dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:38.090827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.090804    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-kube-api-access-sx5vk" pod "91cf765a-5a4f-4f39-828e-0aff2dcb96dc" (UID: "91cf765a-5a4f-4f39-828e-0aff2dcb96dc"). InnerVolumeSpecName "kube-api-access-sx5vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:38.091301 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.091275    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-maas-api-tls" pod "91cf765a-5a4f-4f39-828e-0aff2dcb96dc" (UID: "91cf765a-5a4f-4f39-828e-0aff2dcb96dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:38.091301 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.091289    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-metrics-certs" pod "91cf765a-5a4f-4f39-828e-0aff2dcb96dc" (UID: "91cf765a-5a4f-4f39-828e-0aff2dcb96dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:38.174393 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.174319    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" path="/var/lib/kubelet/pods/162bf7e4-3a15-4b30-a9c6-f816a6327a12/volumes"
Oct 31 02:17:38.174793 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.174777    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a780f315-2f01-4336-958d-c0b59d03e9f3" path="/var/lib/kubelet/pods/a780f315-2f01-4336-958d-c0b59d03e9f3/volumes"
Oct 31 02:17:38.190466 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.190441    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-maas-api-tls\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:38.190466 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.190465    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx5vk\" (UniqueName: \"kubernetes.io/projected/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-kube-api-access-sx5vk\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:38.190659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.190476    5293 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-metrics-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:38.190659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.190485    5293 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf765a-5a4f-4f39-828e-0aff2dcb96dc-openshift-service-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:38.924553 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.924514    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq" event={"ID":"91cf765a-5a4f-4f39-828e-0aff2dcb96dc","Type":"ContainerDied","Data":"49fbdb1c48f86d520aff544df628a0a53a429a63ac9b1e71c1e6ed7d05f9f52b"}
Oct 31 02:17:38.924553 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.924539    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"
Oct 31 02:17:38.925143 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.924587    5293 scope.go:122] "RemoveContainer" containerID="3f9887dc15addf51c66641c64c8d802d6ef9febe05b3645c2d3a5ce0bf5a1e2b"
Oct 31 02:17:38.948490 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.948461    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"]
Oct 31 02:17:38.952203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:38.952180    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c088fd-856f9b49f8-wwtnq"]
Oct 31 02:17:40.175171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.175139    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cf765a-5a4f-4f39-828e-0aff2dcb96dc" path="/var/lib/kubelet/pods/91cf765a-5a4f-4f39-828e-0aff2dcb96dc/volumes"
Oct 31 02:17:40.281560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.281524    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s" podUID="af726e94-0607-4529-a9da-da1d698ff6ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:40.282247 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.282223    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h" podUID="0a86ec52-6276-48bd-9aa3-bf5ad261165e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:40.705813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.705764    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct" podUID="1f67d3a9-5038-4167-8a95-4d23fcd86719" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:40.807120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807084    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"]
Oct 31 02:17:40.807800 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807778    5293 cpu_manager.go:414] "containerMap: removing container" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy"
Oct 31 02:17:40.807800 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807801    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy"
Oct 31 02:17:40.807926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807832    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy"
Oct 31 02:17:40.807926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807838    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy"
Oct 31 02:17:40.807926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807844    5293 cpu_manager.go:414] "containerMap: removing container" podUID="91cf765a-5a4f-4f39-828e-0aff2dcb96dc" containerName="maas-api"
Oct 31 02:17:40.807926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807849    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cf765a-5a4f-4f39-828e-0aff2dcb96dc" containerName="maas-api"
Oct 31 02:17:40.807926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807882    5293 cpu_manager.go:414] "containerMap: removing container" podUID="9634500c-51f5-4687-99f6-a446c6bae152" containerName="revoke-keys"
Oct 31 02:17:40.807926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807887    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9634500c-51f5-4687-99f6-a446c6bae152" containerName="revoke-keys"
Oct 31 02:17:40.808151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807986    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="162bf7e4-3a15-4b30-a9c6-f816a6327a12" containerName="istio-proxy"
Oct 31 02:17:40.808151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.807997    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy"
Oct 31 02:17:40.808151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.808003    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cf765a-5a4f-4f39-828e-0aff2dcb96dc" containerName="maas-api"
Oct 31 02:17:40.808151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.808008    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy"
Oct 31 02:17:40.808151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.808015    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="9634500c-51f5-4687-99f6-a446c6bae152" containerName="revoke-keys"
Oct 31 02:17:40.813030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.813007    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.817926 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.817903    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:17:40.818169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.818144    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:17:40.818279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.818080    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:17:40.818279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.817928    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:17:40.826754 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.826729    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"]
Oct 31 02:17:40.842334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.842293    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.842500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.842396    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6kjt\" (UniqueName: \"kubernetes.io/projected/aa2635f8-bae4-4359-9e29-072f8ef07d21-kube-api-access-d6kjt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.842500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.842451    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.944089 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.944017    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6kjt\" (UniqueName: \"kubernetes.io/projected/aa2635f8-bae4-4359-9e29-072f8ef07d21-kube-api-access-d6kjt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.944276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.944139    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.944387 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.944371    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.944938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.944912    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.947303 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.947280    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:40.955570 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:40.955546    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6kjt\" (UniqueName: \"kubernetes.io/projected/aa2635f8-bae4-4359-9e29-072f8ef07d21-kube-api-access-d6kjt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:41.133230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.132965    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:41.174304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.174260    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:17:41.196965 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.196934    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:17:41.250827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.250757    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af726e94-0607-4529-a9da-da1d698ff6ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af726e94-0607-4529-a9da-da1d698ff6ca-plugins-config-volume\") pod \"af726e94-0607-4529-a9da-da1d698ff6ca\" (UID: \"af726e94-0607-4529-a9da-da1d698ff6ca\") "
Oct 31 02:17:41.250982 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.250835    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a86ec52-6276-48bd-9aa3-bf5ad261165e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a86ec52-6276-48bd-9aa3-bf5ad261165e-plugins-config-volume\") pod \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\" (UID: \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\") "
Oct 31 02:17:41.250982 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.250930    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af726e94-0607-4529-a9da-da1d698ff6ca-kube-api-access-hbp64\" (UniqueName: \"kubernetes.io/projected/af726e94-0607-4529-a9da-da1d698ff6ca-kube-api-access-hbp64\") pod \"af726e94-0607-4529-a9da-da1d698ff6ca\" (UID: \"af726e94-0607-4529-a9da-da1d698ff6ca\") "
Oct 31 02:17:41.251141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.250994    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a86ec52-6276-48bd-9aa3-bf5ad261165e-kube-api-access-rqb5n\" (UniqueName: \"kubernetes.io/projected/0a86ec52-6276-48bd-9aa3-bf5ad261165e-kube-api-access-rqb5n\") pod \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\" (UID: \"0a86ec52-6276-48bd-9aa3-bf5ad261165e\") "
Oct 31 02:17:41.251261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.251239    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af726e94-0607-4529-a9da-da1d698ff6ca-plugins-config-volume" pod "af726e94-0607-4529-a9da-da1d698ff6ca" (UID: "af726e94-0607-4529-a9da-da1d698ff6ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:41.251510 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.251493    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af726e94-0607-4529-a9da-da1d698ff6ca-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:41.252299 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.252277    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a86ec52-6276-48bd-9aa3-bf5ad261165e-plugins-config-volume" pod "0a86ec52-6276-48bd-9aa3-bf5ad261165e" (UID: "0a86ec52-6276-48bd-9aa3-bf5ad261165e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:41.254232 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.254211    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a86ec52-6276-48bd-9aa3-bf5ad261165e-kube-api-access-rqb5n" pod "0a86ec52-6276-48bd-9aa3-bf5ad261165e" (UID: "0a86ec52-6276-48bd-9aa3-bf5ad261165e"). InnerVolumeSpecName "kube-api-access-rqb5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:41.254334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.254255    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af726e94-0607-4529-a9da-da1d698ff6ca-kube-api-access-hbp64" pod "af726e94-0607-4529-a9da-da1d698ff6ca" (UID: "af726e94-0607-4529-a9da-da1d698ff6ca"). InnerVolumeSpecName "kube-api-access-hbp64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:41.353005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.352972    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a86ec52-6276-48bd-9aa3-bf5ad261165e-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:41.353005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.353000    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbp64\" (UniqueName: \"kubernetes.io/projected/af726e94-0607-4529-a9da-da1d698ff6ca-kube-api-access-hbp64\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:41.353005 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.353011    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqb5n\" (UniqueName: \"kubernetes.io/projected/0a86ec52-6276-48bd-9aa3-bf5ad261165e-kube-api-access-rqb5n\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:41.408442 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.408411    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:17:41.440738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.440714    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"]
Oct 31 02:17:41.441678 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:17:41.441649    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2635f8_bae4_4359_9e29_072f8ef07d21.slice/crio-453e5094d09f8f108d0e7027a7f6a15810a890b846a53f33cc593de6fe384d37 WatchSource:0}: Error finding container 453e5094d09f8f108d0e7027a7f6a15810a890b846a53f33cc593de6fe384d37: Status 404 returned error can't find the container with id 453e5094d09f8f108d0e7027a7f6a15810a890b846a53f33cc593de6fe384d37
Oct 31 02:17:41.797937 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:41.797901    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:17:41.972154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.971476    5293 generic.go:356] "Generic (PLEG): container finished" podID="0a86ec52-6276-48bd-9aa3-bf5ad261165e" containerID="868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61" exitCode=0
Oct 31 02:17:41.972154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.971738    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h" event={"ID":"0a86ec52-6276-48bd-9aa3-bf5ad261165e","Type":"ContainerDied","Data":"868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61"}
Oct 31 02:17:41.972154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.971816    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h" event={"ID":"0a86ec52-6276-48bd-9aa3-bf5ad261165e","Type":"ContainerDied","Data":"2501c6f98c24727f7dc3b0f5230e1746f44ef2397e68abf66d5829430a74d296"}
Oct 31 02:17:41.972154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.971844    5293 scope.go:122] "RemoveContainer" containerID="868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61"
Oct 31 02:17:41.972154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.971987    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"
Oct 31 02:17:41.994631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.994547    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp" event={"ID":"aa2635f8-bae4-4359-9e29-072f8ef07d21","Type":"ContainerStarted","Data":"29866cf92de474dc2cf4c34622ec885948f145782e28a8d0f62e9c689a8641f4"}
Oct 31 02:17:41.994996 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:41.994967    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp" event={"ID":"aa2635f8-bae4-4359-9e29-072f8ef07d21","Type":"ContainerStarted","Data":"453e5094d09f8f108d0e7027a7f6a15810a890b846a53f33cc593de6fe384d37"}
Oct 31 02:17:42.010922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.004994    5293 generic.go:356] "Generic (PLEG): container finished" podID="af726e94-0607-4529-a9da-da1d698ff6ca" containerID="89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445" exitCode=0
Oct 31 02:17:42.010922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.005130    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s" event={"ID":"af726e94-0607-4529-a9da-da1d698ff6ca","Type":"ContainerDied","Data":"89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445"}
Oct 31 02:17:42.010922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.005174    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s" event={"ID":"af726e94-0607-4529-a9da-da1d698ff6ca","Type":"ContainerDied","Data":"9ede876f9392fcc53025ac56e2b811c47221180e0465d937051397586b0f3799"}
Oct 31 02:17:42.010922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.005262    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"
Oct 31 02:17:42.010922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.006457    5293 scope.go:122] "RemoveContainer" containerID="868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61"
Oct 31 02:17:42.010922 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:42.006692    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61\": container with ID starting with 868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61 not found: ID does not exist" containerID="868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61"
Oct 31 02:17:42.010922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.006723    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61"} err="failed to get container status \"868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61\": rpc error: code = NotFound desc = could not find container \"868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61\": container with ID starting with 868fc51a0c2e109f0c04b32454b8b3edc2ca7a5bfaa4c7979f98604564b50f61 not found: ID does not exist"
Oct 31 02:17:42.010922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.006744    5293 scope.go:122] "RemoveContainer" containerID="89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445"
Oct 31 02:17:42.021999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.021969    5293 scope.go:122] "RemoveContainer" containerID="89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445"
Oct 31 02:17:42.022444 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:42.022415    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445\": container with ID starting with 89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445 not found: ID does not exist" containerID="89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445"
Oct 31 02:17:42.022567 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.022454    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445"} err="failed to get container status \"89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445\": rpc error: code = NotFound desc = could not find container \"89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445\": container with ID starting with 89ea060e0c3fe06ad37284a6dd0ad0b3ef451e268147bc1a539f4787ffa86445 not found: ID does not exist"
Oct 31 02:17:42.023353 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.023305    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp" podStartSLOduration=2.023290488 podStartE2EDuration="2.023290488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:17:42.019424907 +0000 UTC m=+3419.968073299" watchObservedRunningTime="2026-10-31 02:17:42.023290488 +0000 UTC m=+3419.971938903"
Oct 31 02:17:42.035943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.035913    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"]
Oct 31 02:17:42.039540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.039517    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28e4f3-74bd497f6c-r5p9h"]
Oct 31 02:17:42.049277 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.049252    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"]
Oct 31 02:17:42.055010 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.054991    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28e4f3-9f95fd799-c2x4s"]
Oct 31 02:17:42.207893 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.207856    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a86ec52-6276-48bd-9aa3-bf5ad261165e" path="/var/lib/kubelet/pods/0a86ec52-6276-48bd-9aa3-bf5ad261165e/volumes"
Oct 31 02:17:42.247867 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:42.210402    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af726e94-0607-4529-a9da-da1d698ff6ca" path="/var/lib/kubelet/pods/af726e94-0607-4529-a9da-da1d698ff6ca/volumes"
Oct 31 02:17:43.014978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:43.014947    5293 generic.go:356] "Generic (PLEG): container finished" podID="aa2635f8-bae4-4359-9e29-072f8ef07d21" containerID="29866cf92de474dc2cf4c34622ec885948f145782e28a8d0f62e9c689a8641f4" exitCode=0
Oct 31 02:17:43.015203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:43.015033    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp" event={"ID":"aa2635f8-bae4-4359-9e29-072f8ef07d21","Type":"ContainerDied","Data":"29866cf92de474dc2cf4c34622ec885948f145782e28a8d0f62e9c689a8641f4"}
Oct 31 02:17:44.225903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.225871    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:44.229210 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.229186    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:17:44.306358 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.306320    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f67d3a9-5038-4167-8a95-4d23fcd86719-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f67d3a9-5038-4167-8a95-4d23fcd86719-plugins-config-volume\") pod \"1f67d3a9-5038-4167-8a95-4d23fcd86719\" (UID: \"1f67d3a9-5038-4167-8a95-4d23fcd86719\") "
Oct 31 02:17:44.306558 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.306367    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f67d3a9-5038-4167-8a95-4d23fcd86719-kube-api-access-r5n5m\" (UniqueName: \"kubernetes.io/projected/1f67d3a9-5038-4167-8a95-4d23fcd86719-kube-api-access-r5n5m\") pod \"1f67d3a9-5038-4167-8a95-4d23fcd86719\" (UID: \"1f67d3a9-5038-4167-8a95-4d23fcd86719\") "
Oct 31 02:17:44.306721 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.306692    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2635f8-bae4-4359-9e29-072f8ef07d21-kube-api-access-d6kjt\" (UniqueName: \"kubernetes.io/projected/aa2635f8-bae4-4359-9e29-072f8ef07d21-kube-api-access-d6kjt\") pod \"aa2635f8-bae4-4359-9e29-072f8ef07d21\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") "
Oct 31 02:17:44.306721 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.306715    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f67d3a9-5038-4167-8a95-4d23fcd86719-plugins-config-volume" pod "1f67d3a9-5038-4167-8a95-4d23fcd86719" (UID: "1f67d3a9-5038-4167-8a95-4d23fcd86719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:44.306958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.306748    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-xks-ca\") pod \"aa2635f8-bae4-4359-9e29-072f8ef07d21\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") "
Oct 31 02:17:44.306958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.306785    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-openshift-ca\") pod \"aa2635f8-bae4-4359-9e29-072f8ef07d21\" (UID: \"aa2635f8-bae4-4359-9e29-072f8ef07d21\") "
Oct 31 02:17:44.307390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.307368    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f67d3a9-5038-4167-8a95-4d23fcd86719-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:44.308480 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.308435    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-openshift-ca" pod "aa2635f8-bae4-4359-9e29-072f8ef07d21" (UID: "aa2635f8-bae4-4359-9e29-072f8ef07d21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:44.316689 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.316456    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-xks-ca" pod "aa2635f8-bae4-4359-9e29-072f8ef07d21" (UID: "aa2635f8-bae4-4359-9e29-072f8ef07d21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:44.326212 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.325311    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f67d3a9-5038-4167-8a95-4d23fcd86719-kube-api-access-r5n5m" pod "1f67d3a9-5038-4167-8a95-4d23fcd86719" (UID: "1f67d3a9-5038-4167-8a95-4d23fcd86719"). InnerVolumeSpecName "kube-api-access-r5n5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:44.327782 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.327746    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2635f8-bae4-4359-9e29-072f8ef07d21-kube-api-access-d6kjt" pod "aa2635f8-bae4-4359-9e29-072f8ef07d21" (UID: "aa2635f8-bae4-4359-9e29-072f8ef07d21"). InnerVolumeSpecName "kube-api-access-d6kjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:44.408441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.408405    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:44.408441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.408441    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5n5m\" (UniqueName: \"kubernetes.io/projected/1f67d3a9-5038-4167-8a95-4d23fcd86719-kube-api-access-r5n5m\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:44.408441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.408451    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6kjt\" (UniqueName: \"kubernetes.io/projected/aa2635f8-bae4-4359-9e29-072f8ef07d21-kube-api-access-d6kjt\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:44.408750 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:44.408462    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2635f8-bae4-4359-9e29-072f8ef07d21-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.034509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.034418    5293 generic.go:356] "Generic (PLEG): container finished" podID="1f67d3a9-5038-4167-8a95-4d23fcd86719" containerID="f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723" exitCode=0
Oct 31 02:17:45.034509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.034480    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"
Oct 31 02:17:45.034771 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.034502    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct" event={"ID":"1f67d3a9-5038-4167-8a95-4d23fcd86719","Type":"ContainerDied","Data":"f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723"}
Oct 31 02:17:45.034771 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.034596    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct" event={"ID":"1f67d3a9-5038-4167-8a95-4d23fcd86719","Type":"ContainerDied","Data":"de1a4d530724af1fb3435d208242edd192663f6fc27c89382520f3c44ad03fc3"}
Oct 31 02:17:45.034771 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.034627    5293 scope.go:122] "RemoveContainer" containerID="f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723"
Oct 31 02:17:45.036802 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.036781    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"
Oct 31 02:17:45.036933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.036805    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp" event={"ID":"aa2635f8-bae4-4359-9e29-072f8ef07d21","Type":"ContainerDied","Data":"453e5094d09f8f108d0e7027a7f6a15810a890b846a53f33cc593de6fe384d37"}
Oct 31 02:17:45.036933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.036851    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453e5094d09f8f108d0e7027a7f6a15810a890b846a53f33cc593de6fe384d37"
Oct 31 02:17:45.048171 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.048148    5293 scope.go:122] "RemoveContainer" containerID="f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723"
Oct 31 02:17:45.048438 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:45.048420    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723\": container with ID starting with f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723 not found: ID does not exist" containerID="f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723"
Oct 31 02:17:45.048498 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.048448    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723"} err="failed to get container status \"f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723\": rpc error: code = NotFound desc = could not find container \"f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723\": container with ID starting with f55b5eda0fc7ed585b3c02628f4b344cbdaa4c69db3c305ed6def44f23a7b723 not found: ID does not exist"
Oct 31 02:17:45.061069 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.060361    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"]
Oct 31 02:17:45.066591 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.066563    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91b6b1-7c6b88ff-2zjct"]
Oct 31 02:17:45.683375 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.683338    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"]
Oct 31 02:17:45.683947 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.683594    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy" containerID="cri-o://d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061" gracePeriod=30
Oct 31 02:17:45.928904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.928866    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"]
Oct 31 02:17:45.965125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:45.961278    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ed4dce-d7gvq"]
Oct 31 02:17:46.174390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.174346    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f67d3a9-5038-4167-8a95-4d23fcd86719" path="/var/lib/kubelet/pods/1f67d3a9-5038-4167-8a95-4d23fcd86719/volumes"
Oct 31 02:17:46.174929 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.174905    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9634500c-51f5-4687-99f6-a446c6bae152" path="/var/lib/kubelet/pods/9634500c-51f5-4687-99f6-a446c6bae152/volumes"
Oct 31 02:17:46.218773 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.218699    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"]
Oct 31 02:17:46.219272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219256    5293 cpu_manager.go:414] "containerMap: removing container" podUID="0a86ec52-6276-48bd-9aa3-bf5ad261165e" containerName="payload-processing"
Oct 31 02:17:46.219330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219272    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a86ec52-6276-48bd-9aa3-bf5ad261165e" containerName="payload-processing"
Oct 31 02:17:46.219330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219288    5293 cpu_manager.go:414] "containerMap: removing container" podUID="aa2635f8-bae4-4359-9e29-072f8ef07d21" containerName="revoke-keys"
Oct 31 02:17:46.219330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219294    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2635f8-bae4-4359-9e29-072f8ef07d21" containerName="revoke-keys"
Oct 31 02:17:46.219330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219316    5293 cpu_manager.go:414] "containerMap: removing container" podUID="af726e94-0607-4529-a9da-da1d698ff6ca" containerName="payload-pre-processing"
Oct 31 02:17:46.219330 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219322    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af726e94-0607-4529-a9da-da1d698ff6ca" containerName="payload-pre-processing"
Oct 31 02:17:46.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219340    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy"
Oct 31 02:17:46.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219347    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e177db-8776-47df-96ad-404ec445b4b6" containerName="istio-proxy"
Oct 31 02:17:46.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219355    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1f67d3a9-5038-4167-8a95-4d23fcd86719" containerName="payload-processing"
Oct 31 02:17:46.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219361    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f67d3a9-5038-4167-8a95-4d23fcd86719" containerName="payload-processing"
Oct 31 02:17:46.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219446    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="af726e94-0607-4529-a9da-da1d698ff6ca" containerName="payload-pre-processing"
Oct 31 02:17:46.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219454    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f67d3a9-5038-4167-8a95-4d23fcd86719" containerName="payload-processing"
Oct 31 02:17:46.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219464    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a86ec52-6276-48bd-9aa3-bf5ad261165e" containerName="payload-processing"
Oct 31 02:17:46.219516 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.219475    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2635f8-bae4-4359-9e29-072f8ef07d21" containerName="revoke-keys"
Oct 31 02:17:46.224440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.224420    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.228482 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.228456    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:17:46.228709 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.228522    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:17:46.234372 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.234146    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"]
Oct 31 02:17:46.243689 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.243656    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1537110d-8524-4044-ad98-03a13cab2921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.243838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.243720    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1537110d-8524-4044-ad98-03a13cab2921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.243838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.243801    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5m6\" (UniqueName: \"kubernetes.io/projected/1537110d-8524-4044-ad98-03a13cab2921-kube-api-access-ls5m6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.273682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.273657    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:17:46.273966 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.273948    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:17:46.345448 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.345408    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5m6\" (UniqueName: \"kubernetes.io/projected/1537110d-8524-4044-ad98-03a13cab2921-kube-api-access-ls5m6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.345620 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.345547    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1537110d-8524-4044-ad98-03a13cab2921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.345620 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.345572    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1537110d-8524-4044-ad98-03a13cab2921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.346430 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.346378    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1537110d-8524-4044-ad98-03a13cab2921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.348116 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.348088    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1537110d-8524-4044-ad98-03a13cab2921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.358872 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.358841    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5m6\" (UniqueName: \"kubernetes.io/projected/1537110d-8524-4044-ad98-03a13cab2921-kube-api-access-ls5m6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.543395 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.543304    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:46.862131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:46.855623    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"]
Oct 31 02:17:46.878905 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:17:46.878757    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1537110d_8524_4044_ad98_03a13cab2921.slice/crio-91aa1e63f3e290d0f7b7bc47807e4d30065d40cb9addefc8de84c945025fd223 WatchSource:0}: Error finding container 91aa1e63f3e290d0f7b7bc47807e4d30065d40cb9addefc8de84c945025fd223: Status 404 returned error can't find the container with id 91aa1e63f3e290d0f7b7bc47807e4d30065d40cb9addefc8de84c945025fd223
Oct 31 02:17:47.058456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:47.058419    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp" event={"ID":"1537110d-8524-4044-ad98-03a13cab2921","Type":"ContainerStarted","Data":"91aa1e63f3e290d0f7b7bc47807e4d30065d40cb9addefc8de84c945025fd223"}
Oct 31 02:17:48.067174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:48.067085    5293 generic.go:356] "Generic (PLEG): container finished" podID="1537110d-8524-4044-ad98-03a13cab2921" containerID="1b7f3b6498344aac324976c4618312b8203bd58606d92d03a0c519f17463039d" exitCode=0
Oct 31 02:17:48.067174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:48.067153    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp" event={"ID":"1537110d-8524-4044-ad98-03a13cab2921","Type":"ContainerDied","Data":"1b7f3b6498344aac324976c4618312b8203bd58606d92d03a0c519f17463039d"}
Oct 31 02:17:49.221574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.221543    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:49.302263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.302225    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1537110d-8524-4044-ad98-03a13cab2921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1537110d-8524-4044-ad98-03a13cab2921-maas-api-xks-ca\") pod \"1537110d-8524-4044-ad98-03a13cab2921\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") "
Oct 31 02:17:49.302463 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.302302    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1537110d-8524-4044-ad98-03a13cab2921-kube-api-access-ls5m6\" (UniqueName: \"kubernetes.io/projected/1537110d-8524-4044-ad98-03a13cab2921-kube-api-access-ls5m6\") pod \"1537110d-8524-4044-ad98-03a13cab2921\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") "
Oct 31 02:17:49.302463 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.302453    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1537110d-8524-4044-ad98-03a13cab2921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1537110d-8524-4044-ad98-03a13cab2921-maas-api-openshift-ca\") pod \"1537110d-8524-4044-ad98-03a13cab2921\" (UID: \"1537110d-8524-4044-ad98-03a13cab2921\") "
Oct 31 02:17:49.302884 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.302847    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1537110d-8524-4044-ad98-03a13cab2921-maas-api-openshift-ca" pod "1537110d-8524-4044-ad98-03a13cab2921" (UID: "1537110d-8524-4044-ad98-03a13cab2921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:49.303092 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.303055    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1537110d-8524-4044-ad98-03a13cab2921-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:49.304886 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.304861    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1537110d-8524-4044-ad98-03a13cab2921-maas-api-xks-ca" pod "1537110d-8524-4044-ad98-03a13cab2921" (UID: "1537110d-8524-4044-ad98-03a13cab2921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:49.305192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.305136    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1537110d-8524-4044-ad98-03a13cab2921-kube-api-access-ls5m6" pod "1537110d-8524-4044-ad98-03a13cab2921" (UID: "1537110d-8524-4044-ad98-03a13cab2921"). InnerVolumeSpecName "kube-api-access-ls5m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:49.404374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.404339    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1537110d-8524-4044-ad98-03a13cab2921-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:49.404532 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:49.404379    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls5m6\" (UniqueName: \"kubernetes.io/projected/1537110d-8524-4044-ad98-03a13cab2921-kube-api-access-ls5m6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:50.083580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:50.083549    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"
Oct 31 02:17:50.083781 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:50.083542    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp" event={"ID":"1537110d-8524-4044-ad98-03a13cab2921","Type":"ContainerDied","Data":"91aa1e63f3e290d0f7b7bc47807e4d30065d40cb9addefc8de84c945025fd223"}
Oct 31 02:17:50.083781 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:50.083684    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91aa1e63f3e290d0f7b7bc47807e4d30065d40cb9addefc8de84c945025fd223"
Oct 31 02:17:50.743536 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:50.741757    5293 scope.go:122] "RemoveContainer" containerID="38a4d5ec763a8826708ca97e875e6c31ef0a7dd6b6483e52d03f0e1f3004d805"
Oct 31 02:17:50.805867 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:50.805840    5293 scope.go:122] "RemoveContainer" containerID="add01916dcc5daa07bed8f17fc963c09e9d455dc98bfec5a687d014416b20c6b"
Oct 31 02:17:50.829863 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:50.829452    5293 scope.go:122] "RemoveContainer" containerID="a580021edf03119adaf7527a7876de0e89b6762c460417a138db5374150c8793"
Oct 31 02:17:51.039412 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.039382    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"]
Oct 31 02:17:51.050944 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.050911    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:17:51.094237 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.094200    5293 generic.go:356] "Generic (PLEG): container finished" podID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerID="d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061" exitCode=0
Oct 31 02:17:51.094444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.094263    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" event={"ID":"4cb569eb-af9a-4edf-aa91-a8296743a9b1","Type":"ContainerDied","Data":"d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061"}
Oct 31 02:17:51.094444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.094313    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj" event={"ID":"4cb569eb-af9a-4edf-aa91-a8296743a9b1","Type":"ContainerDied","Data":"1fe0ff84f174b698e99abb6e1828edda94bc0c8079312abff5a4bb4e2a9d8eaa"}
Oct 31 02:17:51.094444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.094386    5293 scope.go:122] "RemoveContainer" containerID="d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061"
Oct 31 02:17:51.094722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.094517    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"
Oct 31 02:17:51.117757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.117728    5293 scope.go:122] "RemoveContainer" containerID="d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061"
Oct 31 02:17:51.118715 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:51.118684    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061\": container with ID starting with d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061 not found: ID does not exist" containerID="d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061"
Oct 31 02:17:51.118854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.118724    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061"} err="failed to get container status \"d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061\": rpc error: code = NotFound desc = could not find container \"d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061\": container with ID starting with d2b2fe0568220aab75701e157ab8afb7c853958dee3dc820324f99b71e448061 not found: ID does not exist"
Oct 31 02:17:51.169409 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169375    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-kube-api-access-x2rd5\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-kube-api-access-x2rd5\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.169585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169442    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-socket\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.169585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169483    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-envoy\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.169585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169522    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-credential-socket\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.169585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169553    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-certs\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.169845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169588    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-token\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-token\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.169845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169621    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-podinfo\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.169845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169684    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istiod-ca-cert\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.169845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169784    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-socket" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.169845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169792    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-data\") pod \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\" (UID: \"4cb569eb-af9a-4edf-aa91-a8296743a9b1\") "
Oct 31 02:17:51.170155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169869    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-certs" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.170155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.169882    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-credential-socket" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.170155 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.170021    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-data" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.170294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.170193    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istiod-ca-cert" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:51.170907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.170870    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.170907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.170900    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.171157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.170916    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.171157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.170932    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.171157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.170948    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.171983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.171960    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-kube-api-access-x2rd5" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "kube-api-access-x2rd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:51.172110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.171985    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-token" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:51.172310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.172283    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-podinfo" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:51.172433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.172416    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-envoy" pod "4cb569eb-af9a-4edf-aa91-a8296743a9b1" (UID: "4cb569eb-af9a-4edf-aa91-a8296743a9b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.272834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.272741    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2rd5\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-kube-api-access-x2rd5\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.272834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.272780    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.272834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.272794    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.272834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.272806    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb569eb-af9a-4edf-aa91-a8296743a9b1-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.431072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.431029    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"]
Oct 31 02:17:51.436605 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:51.436569    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-j97vj"]
Oct 31 02:17:52.174271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:52.174195    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" path="/var/lib/kubelet/pods/4cb569eb-af9a-4edf-aa91-a8296743a9b1/volumes"
Oct 31 02:17:53.284033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.283565    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"]
Oct 31 02:17:53.284033 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.283823    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy" containerID="cri-o://41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7" gracePeriod=30
Oct 31 02:17:53.803854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.803810    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"]
Oct 31 02:17:53.808573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.808545    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-89e955-dr6mp"]
Oct 31 02:17:53.855997 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.855963    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"]
Oct 31 02:17:53.856548 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.856533    5293 cpu_manager.go:414] "containerMap: removing container" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy"
Oct 31 02:17:53.856594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.856550    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy"
Oct 31 02:17:53.856594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.856580    5293 cpu_manager.go:414] "containerMap: removing container" podUID="1537110d-8524-4044-ad98-03a13cab2921" containerName="revoke-keys"
Oct 31 02:17:53.856594 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.856585    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1537110d-8524-4044-ad98-03a13cab2921" containerName="revoke-keys"
Oct 31 02:17:53.856693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.856672    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb569eb-af9a-4edf-aa91-a8296743a9b1" containerName="istio-proxy"
Oct 31 02:17:53.856693 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.856685    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="1537110d-8524-4044-ad98-03a13cab2921" containerName="revoke-keys"
Oct 31 02:17:53.861765 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.861743    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:53.865877 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.865836    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:17:53.865877 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.865863    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:17:53.869539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.869516    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"]
Oct 31 02:17:53.879406 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.879380    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:17:53.879406 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.879378    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:17:53.945167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.945134    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:53.945355 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.945328    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x87b\" (UniqueName: \"kubernetes.io/projected/d5d48681-21d8-45e9-915b-2c09edf48edf-kube-api-access-8x87b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:53.945519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:53.945491    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:54.047443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.047399    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:54.047654 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.047509    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:54.047738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.047661    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x87b\" (UniqueName: \"kubernetes.io/projected/d5d48681-21d8-45e9-915b-2c09edf48edf-kube-api-access-8x87b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:54.048174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.048149    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:54.050245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.050218    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:54.060243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.060144    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x87b\" (UniqueName: \"kubernetes.io/projected/d5d48681-21d8-45e9-915b-2c09edf48edf-kube-api-access-8x87b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:54.173847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.173808    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2635f8-bae4-4359-9e29-072f8ef07d21" path="/var/lib/kubelet/pods/aa2635f8-bae4-4359-9e29-072f8ef07d21/volumes"
Oct 31 02:17:54.182473 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.182452    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:54.471278 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:54.471251    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"]
Oct 31 02:17:54.472450 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:17:54.472419    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d48681_21d8_45e9_915b_2c09edf48edf.slice/crio-c5f08ca520cc241c6ee72a3cde6bb1e951a9631ef8796d13a34ababaae06d9d3 WatchSource:0}: Error finding container c5f08ca520cc241c6ee72a3cde6bb1e951a9631ef8796d13a34ababaae06d9d3: Status 404 returned error can't find the container with id c5f08ca520cc241c6ee72a3cde6bb1e951a9631ef8796d13a34ababaae06d9d3
Oct 31 02:17:55.131195 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:55.131096    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh" event={"ID":"d5d48681-21d8-45e9-915b-2c09edf48edf","Type":"ContainerStarted","Data":"0970715b0a3cfb5c81029b87473b25236a9d8c844c2c3a75d686d7204f189c37"}
Oct 31 02:17:55.131384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:55.131203    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh" event={"ID":"d5d48681-21d8-45e9-915b-2c09edf48edf","Type":"ContainerStarted","Data":"c5f08ca520cc241c6ee72a3cde6bb1e951a9631ef8796d13a34ababaae06d9d3"}
Oct 31 02:17:55.150452 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:55.150407    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh" podStartSLOduration=2.15039023 podStartE2EDuration="2.15039023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:17:55.146907505 +0000 UTC m=+3433.095555896" watchObservedRunningTime="2026-10-31 02:17:55.15039023 +0000 UTC m=+3433.099038622"
Oct 31 02:17:56.040530 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.040486    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn" podUID="240f0d9e-a0f1-4dd3-969e-287516d9aa97" containerName="payload-pre-processing" containerID="cri-o://5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106" gracePeriod=25
Oct 31 02:17:56.141002 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.140964    5293 generic.go:356] "Generic (PLEG): container finished" podID="d5d48681-21d8-45e9-915b-2c09edf48edf" containerID="0970715b0a3cfb5c81029b87473b25236a9d8c844c2c3a75d686d7204f189c37" exitCode=0
Oct 31 02:17:56.141218 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.141060    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh" event={"ID":"d5d48681-21d8-45e9-915b-2c09edf48edf","Type":"ContainerDied","Data":"0970715b0a3cfb5c81029b87473b25236a9d8c844c2c3a75d686d7204f189c37"}
Oct 31 02:17:56.355409 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.355385    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 02:17:56.484525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.484497    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240f0d9e-a0f1-4dd3-969e-287516d9aa97-kube-api-access-gfcdl\" (UniqueName: \"kubernetes.io/projected/240f0d9e-a0f1-4dd3-969e-287516d9aa97-kube-api-access-gfcdl\") pod \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\" (UID: \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\") "
Oct 31 02:17:56.484828 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.484808    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240f0d9e-a0f1-4dd3-969e-287516d9aa97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/240f0d9e-a0f1-4dd3-969e-287516d9aa97-plugins-config-volume\") pod \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\" (UID: \"240f0d9e-a0f1-4dd3-969e-287516d9aa97\") "
Oct 31 02:17:56.485347 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.485317    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240f0d9e-a0f1-4dd3-969e-287516d9aa97-plugins-config-volume" pod "240f0d9e-a0f1-4dd3-969e-287516d9aa97" (UID: "240f0d9e-a0f1-4dd3-969e-287516d9aa97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:56.486860 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.486833    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240f0d9e-a0f1-4dd3-969e-287516d9aa97-kube-api-access-gfcdl" pod "240f0d9e-a0f1-4dd3-969e-287516d9aa97" (UID: "240f0d9e-a0f1-4dd3-969e-287516d9aa97"). InnerVolumeSpecName "kube-api-access-gfcdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:56.586131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.586058    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfcdl\" (UniqueName: \"kubernetes.io/projected/240f0d9e-a0f1-4dd3-969e-287516d9aa97-kube-api-access-gfcdl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:56.586131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:56.586087    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/240f0d9e-a0f1-4dd3-969e-287516d9aa97-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:57.148844 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.148807    5293 generic.go:356] "Generic (PLEG): container finished" podID="240f0d9e-a0f1-4dd3-969e-287516d9aa97" containerID="5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106" exitCode=0
Oct 31 02:17:57.149272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.148856    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"
Oct 31 02:17:57.149272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.148898    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn" event={"ID":"240f0d9e-a0f1-4dd3-969e-287516d9aa97","Type":"ContainerDied","Data":"5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106"}
Oct 31 02:17:57.149272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.148977    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn" event={"ID":"240f0d9e-a0f1-4dd3-969e-287516d9aa97","Type":"ContainerDied","Data":"4bbbb9ec6466aaf7384fc8dfdbc7f19c8c1fa6fb3f8e9e798a6b582488242957"}
Oct 31 02:17:57.149272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.148999    5293 scope.go:122] "RemoveContainer" containerID="5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106"
Oct 31 02:17:57.160796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.160773    5293 scope.go:122] "RemoveContainer" containerID="5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106"
Oct 31 02:17:57.161215 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:57.161184    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106\": container with ID starting with 5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106 not found: ID does not exist" containerID="5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106"
Oct 31 02:17:57.161324 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.161227    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106"} err="failed to get container status \"5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106\": rpc error: code = NotFound desc = could not find container \"5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106\": container with ID starting with 5fc15e3cdffda79158424095a10326c0e063fa831b03631e7b20f202cd20f106 not found: ID does not exist"
Oct 31 02:17:57.175188 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.175162    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"]
Oct 31 02:17:57.180417 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.180392    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-89e955-58f65868c9-nwtnn"]
Oct 31 02:17:57.313261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.313238    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:57.398637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.398603    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-xks-ca\") pod \"d5d48681-21d8-45e9-915b-2c09edf48edf\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") "
Oct 31 02:17:57.398803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.398677    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d48681-21d8-45e9-915b-2c09edf48edf-kube-api-access-8x87b\" (UniqueName: \"kubernetes.io/projected/d5d48681-21d8-45e9-915b-2c09edf48edf-kube-api-access-8x87b\") pod \"d5d48681-21d8-45e9-915b-2c09edf48edf\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") "
Oct 31 02:17:57.398902 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.398838    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-openshift-ca\") pod \"d5d48681-21d8-45e9-915b-2c09edf48edf\" (UID: \"d5d48681-21d8-45e9-915b-2c09edf48edf\") "
Oct 31 02:17:57.399194 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.399171    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-openshift-ca" pod "d5d48681-21d8-45e9-915b-2c09edf48edf" (UID: "d5d48681-21d8-45e9-915b-2c09edf48edf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:57.399381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.399364    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:57.400841 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.400818    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-xks-ca" pod "d5d48681-21d8-45e9-915b-2c09edf48edf" (UID: "d5d48681-21d8-45e9-915b-2c09edf48edf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:57.400943 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.400843    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d48681-21d8-45e9-915b-2c09edf48edf-kube-api-access-8x87b" pod "d5d48681-21d8-45e9-915b-2c09edf48edf" (UID: "d5d48681-21d8-45e9-915b-2c09edf48edf"). InnerVolumeSpecName "kube-api-access-8x87b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:57.501354 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.501318    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d48681-21d8-45e9-915b-2c09edf48edf-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:57.501513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.501357    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x87b\" (UniqueName: \"kubernetes.io/projected/d5d48681-21d8-45e9-915b-2c09edf48edf-kube-api-access-8x87b\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:57.755199 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.755110    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"]
Oct 31 02:17:57.764355 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:57.764247    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7a850e-mwgmp"]
Oct 31 02:17:58.158259 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.158231    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"
Oct 31 02:17:58.158712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.158269    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh" event={"ID":"d5d48681-21d8-45e9-915b-2c09edf48edf","Type":"ContainerDied","Data":"c5f08ca520cc241c6ee72a3cde6bb1e951a9631ef8796d13a34ababaae06d9d3"}
Oct 31 02:17:58.158712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.158326    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5f08ca520cc241c6ee72a3cde6bb1e951a9631ef8796d13a34ababaae06d9d3"
Oct 31 02:17:58.173944 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.173915    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1537110d-8524-4044-ad98-03a13cab2921" path="/var/lib/kubelet/pods/1537110d-8524-4044-ad98-03a13cab2921/volumes"
Oct 31 02:17:58.174507 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.174491    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240f0d9e-a0f1-4dd3-969e-287516d9aa97" path="/var/lib/kubelet/pods/240f0d9e-a0f1-4dd3-969e-287516d9aa97/volumes"
Oct 31 02:17:58.556336 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.556312    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 02:17:58.721934 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.721861    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-credential-socket\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.721934 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.721907    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-envoy\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.722190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.721985    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-token\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-token\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.722190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722016    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-podinfo\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.722190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722090    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istiod-ca-cert\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.722374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722196    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-certs\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.722374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722205    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-credential-socket" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:58.722374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722228    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-socket\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.722374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722247    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-data\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.722374 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722267    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-kube-api-access-gmbb2\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-kube-api-access-gmbb2\") pod \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\" (UID: \"e2088de2-2f63-4bc6-8e34-6fdaaa69593b\") "
Oct 31 02:17:58.722659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722530    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-socket" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:58.722659 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722632    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istiod-ca-cert" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:58.722794 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722769    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-data" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:58.722858 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.722830    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-certs" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:58.723230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.723077    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:58.723230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.723096    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:58.723230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.723105    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:58.723230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.723114    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:58.723230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.723122    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:58.724394 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.724366    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-token" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:58.724823 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.724796    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-kube-api-access-gmbb2" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "kube-api-access-gmbb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:58.724947 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.724845    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-envoy" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:58.725132 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.725111    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-podinfo" pod "e2088de2-2f63-4bc6-8e34-6fdaaa69593b" (UID: "e2088de2-2f63-4bc6-8e34-6fdaaa69593b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:58.824297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.824265    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:58.824297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.824295    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:58.824297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.824306    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbb2\" (UniqueName: \"kubernetes.io/projected/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-kube-api-access-gmbb2\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:58.824544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:58.824315    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2088de2-2f63-4bc6-8e34-6fdaaa69593b-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:59.172156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.171637    5293 generic.go:356] "Generic (PLEG): container finished" podID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerID="41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7" exitCode=0
Oct 31 02:17:59.172156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.171833    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"
Oct 31 02:17:59.172156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.171849    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" event={"ID":"e2088de2-2f63-4bc6-8e34-6fdaaa69593b","Type":"ContainerDied","Data":"41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7"}
Oct 31 02:17:59.172156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.171947    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m" event={"ID":"e2088de2-2f63-4bc6-8e34-6fdaaa69593b","Type":"ContainerDied","Data":"7c9599532e2c6e68ed6ef05cad797c2254245c4b31bc8a3a5c1d1cf7dc13d1ac"}
Oct 31 02:17:59.172156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.171991    5293 scope.go:122] "RemoveContainer" containerID="41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7"
Oct 31 02:17:59.199904 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.199871    5293 scope.go:122] "RemoveContainer" containerID="1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0"
Oct 31 02:17:59.202274 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.202244    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"]
Oct 31 02:17:59.205850 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.205830    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed4dce-openshift-default-7f55785c6f-9tt5m"]
Oct 31 02:17:59.212245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.212224    5293 scope.go:122] "RemoveContainer" containerID="41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7"
Oct 31 02:17:59.212585 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:59.212552    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7\": container with ID starting with 41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7 not found: ID does not exist" containerID="41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7"
Oct 31 02:17:59.212674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.212599    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7"} err="failed to get container status \"41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7\": rpc error: code = NotFound desc = could not find container \"41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7\": container with ID starting with 41a632d49a217eba23c7e45588711342906e8e6e7620d510fd440c5f916b8cb7 not found: ID does not exist"
Oct 31 02:17:59.212674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.212625    5293 scope.go:122] "RemoveContainer" containerID="1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0"
Oct 31 02:17:59.212909 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:17:59.212891    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0\": container with ID starting with 1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0 not found: ID does not exist" containerID="1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0"
Oct 31 02:17:59.212958 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:17:59.212916    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0"} err="failed to get container status \"1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0\": rpc error: code = NotFound desc = could not find container \"1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0\": container with ID starting with 1f247119bcef8d3d1f77a70bff5c9de9b3d6cf9dc9cb5b14f49a1b9264eb55f0 not found: ID does not exist"
Oct 31 02:18:00.173636 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:00.173601    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" path="/var/lib/kubelet/pods/e2088de2-2f63-4bc6-8e34-6fdaaa69593b/volumes"
Oct 31 02:18:02.293255 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293218    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"]
Oct 31 02:18:02.293752 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293739    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy"
Oct 31 02:18:02.293813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293754    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy"
Oct 31 02:18:02.293813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293778    5293 cpu_manager.go:414] "containerMap: removing container" podUID="240f0d9e-a0f1-4dd3-969e-287516d9aa97" containerName="payload-pre-processing"
Oct 31 02:18:02.293813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293784    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240f0d9e-a0f1-4dd3-969e-287516d9aa97" containerName="payload-pre-processing"
Oct 31 02:18:02.293813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293795    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d5d48681-21d8-45e9-915b-2c09edf48edf" containerName="revoke-keys"
Oct 31 02:18:02.293813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293808    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d48681-21d8-45e9-915b-2c09edf48edf" containerName="revoke-keys"
Oct 31 02:18:02.293980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293897    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy"
Oct 31 02:18:02.293980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293906    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="240f0d9e-a0f1-4dd3-969e-287516d9aa97" containerName="payload-pre-processing"
Oct 31 02:18:02.293980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293914    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d48681-21d8-45e9-915b-2c09edf48edf" containerName="revoke-keys"
Oct 31 02:18:02.293980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.293924    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy"
Oct 31 02:18:02.303970 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.303943    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.309428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.308118    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"]
Oct 31 02:18:02.309428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.308731    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:18:02.309428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.308803    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:18:02.309428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.309307    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:18:02.309428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.309365    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:18:02.380998 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.380960    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.380998 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.381000    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6c5n\" (UniqueName: \"kubernetes.io/projected/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-kube-api-access-r6c5n\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.381240 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.381195    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.482238 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.482198    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.482449 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.482347    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.482449 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.482377    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6c5n\" (UniqueName: \"kubernetes.io/projected/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-kube-api-access-r6c5n\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.482833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.482814    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.485067 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.485032    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.494302 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.494278    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6c5n\" (UniqueName: \"kubernetes.io/projected/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-kube-api-access-r6c5n\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.623107 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.622993    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:02.695669 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.695639    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:18:02.932682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:02.932633    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"]
Oct 31 02:18:03.202011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:03.201946    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556" event={"ID":"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa","Type":"ContainerStarted","Data":"c185aee837a5b290b051de3ed0d2a4c6743243b1078bfa577dd9ee556dac3c66"}
Oct 31 02:18:04.011280 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:04.011250    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"]
Oct 31 02:18:04.212895 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:04.212864    5293 generic.go:356] "Generic (PLEG): container finished" podID="f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa" containerID="9df7d5660e5360bd6ee3851cbc468991e8d8a0843966a87800004090a58a34c0" exitCode=0
Oct 31 02:18:04.213150 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:04.212955    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556" event={"ID":"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa","Type":"ContainerDied","Data":"9df7d5660e5360bd6ee3851cbc468991e8d8a0843966a87800004090a58a34c0"}
Oct 31 02:18:05.248402 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.248371    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"]
Oct 31 02:18:05.248887 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.248602    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" containerID="cri-o://1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2" gracePeriod=30
Oct 31 02:18:05.464062 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.461232    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"]
Oct 31 02:18:05.480654 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.479333    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2e6bfd-nz9lh"]
Oct 31 02:18:05.562427 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.562397    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:05.680061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.679605    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-kube-api-access-r6c5n\" (UniqueName: \"kubernetes.io/projected/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-kube-api-access-r6c5n\") pod \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") "
Oct 31 02:18:05.680061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.679849    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-xks-ca\") pod \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") "
Oct 31 02:18:05.680061 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.679943    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-openshift-ca\") pod \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\" (UID: \"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa\") "
Oct 31 02:18:05.686169 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.684007    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-openshift-ca" pod "f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa" (UID: "f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:05.691323 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.686807    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-kube-api-access-r6c5n" pod "f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa" (UID: "f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa"). InnerVolumeSpecName "kube-api-access-r6c5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:05.696129 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.696098    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-xks-ca" pod "f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa" (UID: "f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:05.767709 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.767639    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"]
Oct 31 02:18:05.768190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.768176    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy"
Oct 31 02:18:05.768190 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.768190    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2088de2-2f63-4bc6-8e34-6fdaaa69593b" containerName="istio-proxy"
Oct 31 02:18:05.768345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.768206    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa" containerName="revoke-keys"
Oct 31 02:18:05.768345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.768211    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa" containerName="revoke-keys"
Oct 31 02:18:05.768345 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.768301    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa" containerName="revoke-keys"
Oct 31 02:18:05.771770 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.771739    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.782422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.782393    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:05.782540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.782429    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:05.782540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.782448    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6c5n\" (UniqueName: \"kubernetes.io/projected/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa-kube-api-access-r6c5n\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:05.782667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.782610    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"]
Oct 31 02:18:05.883667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.883637    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.883667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.883671    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6fv\" (UniqueName: \"kubernetes.io/projected/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-kube-api-access-nc6fv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.883878 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.883754    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.985458 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.985425    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.985458 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.985462    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6fv\" (UniqueName: \"kubernetes.io/projected/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-kube-api-access-nc6fv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.985691 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.985501    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.986195 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.986169    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.987938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.987915    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:05.998080 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:05.998034    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6fv\" (UniqueName: \"kubernetes.io/projected/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-kube-api-access-nc6fv\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:06.089990 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:06.089912    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:06.174674 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:06.174637    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d48681-21d8-45e9-915b-2c09edf48edf" path="/var/lib/kubelet/pods/d5d48681-21d8-45e9-915b-2c09edf48edf/volumes"
Oct 31 02:18:06.233759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:06.233726    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"
Oct 31 02:18:06.233971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:06.233721    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556" event={"ID":"f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa","Type":"ContainerDied","Data":"c185aee837a5b290b051de3ed0d2a4c6743243b1078bfa577dd9ee556dac3c66"}
Oct 31 02:18:06.233971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:06.233876    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c185aee837a5b290b051de3ed0d2a4c6743243b1078bfa577dd9ee556dac3c66"
Oct 31 02:18:06.376106 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:06.376078    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"]
Oct 31 02:18:06.377678 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:18:06.377648    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59066d6_007e_42f2_b6a0_0f8c7e5b2403.slice/crio-97b806513de0a28299296fe160055fd9c4142429d6b95a61bf59eeed0a6ac4bf WatchSource:0}: Error finding container 97b806513de0a28299296fe160055fd9c4142429d6b95a61bf59eeed0a6ac4bf: Status 404 returned error can't find the container with id 97b806513de0a28299296fe160055fd9c4142429d6b95a61bf59eeed0a6ac4bf
Oct 31 02:18:06.893344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:06.893305    5293 patch_prober.go:29] interesting pod/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:18:06.893519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:06.893366    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:18:07.074273 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:07.072244    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"]
Oct 31 02:18:07.087725 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:07.087690    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"]
Oct 31 02:18:07.091652 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:07.091625    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"]
Oct 31 02:18:07.259161 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:07.259127    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt" event={"ID":"e59066d6-007e-42f2-b6a0-0f8c7e5b2403","Type":"ContainerStarted","Data":"d1ce5d0c7615698efc06f81892923f549197b938f00b87350ef6eff90f58ae15"}
Oct 31 02:18:07.259338 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:07.259185    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt" event={"ID":"e59066d6-007e-42f2-b6a0-0f8c7e5b2403","Type":"ContainerStarted","Data":"97b806513de0a28299296fe160055fd9c4142429d6b95a61bf59eeed0a6ac4bf"}
Oct 31 02:18:07.278587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:07.278545    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt" podStartSLOduration=2.278531572 podStartE2EDuration="2.278531572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:18:07.276018931 +0000 UTC m=+3445.224667373" watchObservedRunningTime="2026-10-31 02:18:07.278531572 +0000 UTC m=+3445.227179964"
Oct 31 02:18:08.267490 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:08.267460    5293 generic.go:356] "Generic (PLEG): container finished" podID="e59066d6-007e-42f2-b6a0-0f8c7e5b2403" containerID="d1ce5d0c7615698efc06f81892923f549197b938f00b87350ef6eff90f58ae15" exitCode=0
Oct 31 02:18:08.267921 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:08.267551    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt" event={"ID":"e59066d6-007e-42f2-b6a0-0f8c7e5b2403","Type":"ContainerDied","Data":"d1ce5d0c7615698efc06f81892923f549197b938f00b87350ef6eff90f58ae15"}
Oct 31 02:18:09.011785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.011746    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l" podUID="945eac35-ca1c-4d53-974a-bc2d7df38685" containerName="payload-pre-processing" containerID="cri-o://614f0b02914173bc4cce4cb4fc37a5fb143c09acd1a06fc77c285b58a94fd92c" gracePeriod=25
Oct 31 02:18:09.277745 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.277711    5293 generic.go:356] "Generic (PLEG): container finished" podID="945eac35-ca1c-4d53-974a-bc2d7df38685" containerID="614f0b02914173bc4cce4cb4fc37a5fb143c09acd1a06fc77c285b58a94fd92c" exitCode=0
Oct 31 02:18:09.278119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.277783    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l" event={"ID":"945eac35-ca1c-4d53-974a-bc2d7df38685","Type":"ContainerDied","Data":"614f0b02914173bc4cce4cb4fc37a5fb143c09acd1a06fc77c285b58a94fd92c"}
Oct 31 02:18:09.278119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.277857    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l" event={"ID":"945eac35-ca1c-4d53-974a-bc2d7df38685","Type":"ContainerDied","Data":"1db8d5c55152b82e1d6edc26d474ef21fd401a822ac8e63cdb5b38a85cae3149"}
Oct 31 02:18:09.278119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.277880    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db8d5c55152b82e1d6edc26d474ef21fd401a822ac8e63cdb5b38a85cae3149"
Oct 31 02:18:09.293897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.293868    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 02:18:09.389931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.385824    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945eac35-ca1c-4d53-974a-bc2d7df38685-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945eac35-ca1c-4d53-974a-bc2d7df38685-plugins-config-volume\") pod \"945eac35-ca1c-4d53-974a-bc2d7df38685\" (UID: \"945eac35-ca1c-4d53-974a-bc2d7df38685\") "
Oct 31 02:18:09.389931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.385885    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945eac35-ca1c-4d53-974a-bc2d7df38685-kube-api-access-vvm5q\" (UniqueName: \"kubernetes.io/projected/945eac35-ca1c-4d53-974a-bc2d7df38685-kube-api-access-vvm5q\") pod \"945eac35-ca1c-4d53-974a-bc2d7df38685\" (UID: \"945eac35-ca1c-4d53-974a-bc2d7df38685\") "
Oct 31 02:18:09.389931 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.387604    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945eac35-ca1c-4d53-974a-bc2d7df38685-plugins-config-volume" pod "945eac35-ca1c-4d53-974a-bc2d7df38685" (UID: "945eac35-ca1c-4d53-974a-bc2d7df38685"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:09.391819 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.391332    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945eac35-ca1c-4d53-974a-bc2d7df38685-kube-api-access-vvm5q" pod "945eac35-ca1c-4d53-974a-bc2d7df38685" (UID: "945eac35-ca1c-4d53-974a-bc2d7df38685"). InnerVolumeSpecName "kube-api-access-vvm5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:09.487156 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.487132    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:09.487851 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.487831    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945eac35-ca1c-4d53-974a-bc2d7df38685-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:09.487922 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.487853    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvm5q\" (UniqueName: \"kubernetes.io/projected/945eac35-ca1c-4d53-974a-bc2d7df38685-kube-api-access-vvm5q\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:09.589297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.589203    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-kube-api-access-nc6fv\" (UniqueName: \"kubernetes.io/projected/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-kube-api-access-nc6fv\") pod \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") "
Oct 31 02:18:09.589469 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.589314    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-xks-ca\") pod \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") "
Oct 31 02:18:09.589469 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.589404    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-openshift-ca\") pod \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\" (UID: \"e59066d6-007e-42f2-b6a0-0f8c7e5b2403\") "
Oct 31 02:18:09.589827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.589787    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-openshift-ca" pod "e59066d6-007e-42f2-b6a0-0f8c7e5b2403" (UID: "e59066d6-007e-42f2-b6a0-0f8c7e5b2403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:09.590281 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.590260    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:09.591529 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.591508    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-xks-ca" pod "e59066d6-007e-42f2-b6a0-0f8c7e5b2403" (UID: "e59066d6-007e-42f2-b6a0-0f8c7e5b2403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:09.591618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.591534    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-kube-api-access-nc6fv" pod "e59066d6-007e-42f2-b6a0-0f8c7e5b2403" (UID: "e59066d6-007e-42f2-b6a0-0f8c7e5b2403"). InnerVolumeSpecName "kube-api-access-nc6fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:09.691486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.691455    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc6fv\" (UniqueName: \"kubernetes.io/projected/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-kube-api-access-nc6fv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:09.691486 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:09.691482    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59066d6-007e-42f2-b6a0-0f8c7e5b2403-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.289119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.289087    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"
Oct 31 02:18:10.289119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.289092    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt" event={"ID":"e59066d6-007e-42f2-b6a0-0f8c7e5b2403","Type":"ContainerDied","Data":"97b806513de0a28299296fe160055fd9c4142429d6b95a61bf59eeed0a6ac4bf"}
Oct 31 02:18:10.289497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.289176    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97b806513de0a28299296fe160055fd9c4142429d6b95a61bf59eeed0a6ac4bf"
Oct 31 02:18:10.289497 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.289252    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"
Oct 31 02:18:10.350419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.350390    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"]
Oct 31 02:18:10.357508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.357480    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2e6bfd-789f97d447-w877l"]
Oct 31 02:18:10.528383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.528327    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 02:18:10.601460 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601423    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-kube-api-access-cfpzm\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-kube-api-access-cfpzm\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.601651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601478    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-certs\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.601651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601540    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-token\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-token\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.601651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601564    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a87c65c-8925-4bf1-952c-f380e42e2023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a87c65c-8925-4bf1-952c-f380e42e2023-istiod-ca-cert\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.601651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601594    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-podinfo\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.601651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601638    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-data\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.601955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601761    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-credential-socket\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.601955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601855    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-envoy\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.601955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601871    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-certs" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:10.601955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.601926    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-socket\") pod \"8a87c65c-8925-4bf1-952c-f380e42e2023\" (UID: \"8a87c65c-8925-4bf1-952c-f380e42e2023\") "
Oct 31 02:18:10.602737 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.602704    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-credential-socket" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:10.603013 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.602961    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-data" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:10.603295 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.603268    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-socket" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:10.603443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.603422    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a87c65c-8925-4bf1-952c-f380e42e2023-istiod-ca-cert" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:10.603830 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.603808    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.603925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.603835    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.603925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.603853    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.603925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.603868    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.603925 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.603881    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a87c65c-8925-4bf1-952c-f380e42e2023-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.604408 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.604383    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-envoy" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:10.604699 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.604671    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-token" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:10.604793 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.604727    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-kube-api-access-cfpzm" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "kube-api-access-cfpzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:10.605320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.605301    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-podinfo" pod "8a87c65c-8925-4bf1-952c-f380e42e2023" (UID: "8a87c65c-8925-4bf1-952c-f380e42e2023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:10.704908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.704874    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.704908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.704903    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfpzm\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-kube-api-access-cfpzm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.704908 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.704914    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:10.705182 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:10.704923    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a87c65c-8925-4bf1-952c-f380e42e2023-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:11.298784 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.298749    5293 generic.go:356] "Generic (PLEG): container finished" podID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerID="1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2" exitCode=0
Oct 31 02:18:11.299334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.298807    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"
Oct 31 02:18:11.299334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.298842    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" event={"ID":"8a87c65c-8925-4bf1-952c-f380e42e2023","Type":"ContainerDied","Data":"1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2"}
Oct 31 02:18:11.299334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.298911    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5" event={"ID":"8a87c65c-8925-4bf1-952c-f380e42e2023","Type":"ContainerDied","Data":"98f4a7bd3b41d9b2a74862dfa22eb3b8bf38c9cc27f462e74b6b37fcc3910fd8"}
Oct 31 02:18:11.299334 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.298942    5293 scope.go:122] "RemoveContainer" containerID="1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2"
Oct 31 02:18:11.314473 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.314457    5293 scope.go:122] "RemoveContainer" containerID="d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e"
Oct 31 02:18:11.324627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.324540    5293 scope.go:122] "RemoveContainer" containerID="1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2"
Oct 31 02:18:11.324883 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:11.324856    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2\": container with ID starting with 1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2 not found: ID does not exist" containerID="1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2"
Oct 31 02:18:11.324978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.324896    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2"} err="failed to get container status \"1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2\": rpc error: code = NotFound desc = could not find container \"1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2\": container with ID starting with 1f2666ef8cae8814ffa29f1fa5258066c6946b7265b3e5498180d1a8982de1c2 not found: ID does not exist"
Oct 31 02:18:11.324978 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.324919    5293 scope.go:122] "RemoveContainer" containerID="d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e"
Oct 31 02:18:11.325511 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:11.325482    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e\": container with ID starting with d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e not found: ID does not exist" containerID="d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e"
Oct 31 02:18:11.325563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.325522    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e"} err="failed to get container status \"d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e\": rpc error: code = NotFound desc = could not find container \"d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e\": container with ID starting with d6d90fb9d770bd92a4a56fddbcd7df4a2f62d4271c00134c51b8c53969ae414e not found: ID does not exist"
Oct 31 02:18:11.331269 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.331244    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"]
Oct 31 02:18:11.338648 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:11.338628    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7a850e-openshift-default-848ffd8d8f-txcb5"]
Oct 31 02:18:11.777292 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:11.777246    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:18:12.174597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:12.174566    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" path="/var/lib/kubelet/pods/8a87c65c-8925-4bf1-952c-f380e42e2023/volumes"
Oct 31 02:18:12.175153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:12.175129    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945eac35-ca1c-4d53-974a-bc2d7df38685" path="/var/lib/kubelet/pods/945eac35-ca1c-4d53-974a-bc2d7df38685/volumes"
Oct 31 02:18:12.437140 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:12.437058    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"]
Oct 31 02:18:12.454060 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:12.450471    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"]
Oct 31 02:18:13.121397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.121363    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"]
Oct 31 02:18:13.121634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.121596    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy" containerID="cri-o://02bfbf1726a2eecb1b335896818cdfc46c1d8e5091113a135575ef92fc0da4fa" gracePeriod=30
Oct 31 02:18:13.467100 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.467030    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:18:13.942462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.942420    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"]
Oct 31 02:18:13.942899 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.942885    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy"
Oct 31 02:18:13.942899 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.942898    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy"
Oct 31 02:18:13.943074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.942918    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e59066d6-007e-42f2-b6a0-0f8c7e5b2403" containerName="revoke-keys"
Oct 31 02:18:13.943074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.942924    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59066d6-007e-42f2-b6a0-0f8c7e5b2403" containerName="revoke-keys"
Oct 31 02:18:13.943074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.942938    5293 cpu_manager.go:414] "containerMap: removing container" podUID="945eac35-ca1c-4d53-974a-bc2d7df38685" containerName="payload-pre-processing"
Oct 31 02:18:13.943074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.942943    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945eac35-ca1c-4d53-974a-bc2d7df38685" containerName="payload-pre-processing"
Oct 31 02:18:13.943074 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.943071    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy"
Oct 31 02:18:13.943397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.943084    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="945eac35-ca1c-4d53-974a-bc2d7df38685" containerName="payload-pre-processing"
Oct 31 02:18:13.943397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.943107    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59066d6-007e-42f2-b6a0-0f8c7e5b2403" containerName="revoke-keys"
Oct 31 02:18:13.948553 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.948533    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:13.953068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.953026    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:18:13.954888 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.954866    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:18:13.958028 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.957817    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"]
Oct 31 02:18:13.962243 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.962220    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:18:13.962332 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:13.962257    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:18:14.050580 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.050545    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c864b\" (UniqueName: \"kubernetes.io/projected/283c48f8-8987-458f-8d6e-07ea7ab01754-kube-api-access-c864b\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.050785 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.050623    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.050833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.050787    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.152183 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.152144    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.152384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.152247    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c864b\" (UniqueName: \"kubernetes.io/projected/283c48f8-8987-458f-8d6e-07ea7ab01754-kube-api-access-c864b\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.152384 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.152319    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.152809 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.152783    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.154903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.154885    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.163857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.163835    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c864b\" (UniqueName: \"kubernetes.io/projected/283c48f8-8987-458f-8d6e-07ea7ab01754-kube-api-access-c864b\") pod \"maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.265286 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.265194    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:14.549631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:14.549607    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"]
Oct 31 02:18:14.552398 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:18:14.552352    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283c48f8_8987_458f_8d6e_07ea7ab01754.slice/crio-c2cbe9c76d9afef8457a76ba5b7f10d118e0c3b000680e90096cb522d2070fa2 WatchSource:0}: Error finding container c2cbe9c76d9afef8457a76ba5b7f10d118e0c3b000680e90096cb522d2070fa2: Status 404 returned error can't find the container with id c2cbe9c76d9afef8457a76ba5b7f10d118e0c3b000680e90096cb522d2070fa2
Oct 31 02:18:15.339344 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.339303    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx" event={"ID":"283c48f8-8987-458f-8d6e-07ea7ab01754","Type":"ContainerStarted","Data":"dd8fc8ffebe3b1229ff240ab574ef5ca6d96ba772406b5d3a08f7f4ebffd32b6"}
Oct 31 02:18:15.339559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.339374    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx" event={"ID":"283c48f8-8987-458f-8d6e-07ea7ab01754","Type":"ContainerStarted","Data":"c2cbe9c76d9afef8457a76ba5b7f10d118e0c3b000680e90096cb522d2070fa2"}
Oct 31 02:18:15.356957 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.356899    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx" podStartSLOduration=2.356880468 podStartE2EDuration="2.356880468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:18:15.355239314 +0000 UTC m=+3453.303887705" watchObservedRunningTime="2026-10-31 02:18:15.356880468 +0000 UTC m=+3453.305528861"
Oct 31 02:18:15.846759 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.846720    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"]
Oct 31 02:18:15.847462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.847431    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy"
Oct 31 02:18:15.847462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.847459    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy"
Oct 31 02:18:15.847616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.847601    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a87c65c-8925-4bf1-952c-f380e42e2023" containerName="istio-proxy"
Oct 31 02:18:15.851548 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.851515    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.862681 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.862655    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"]
Oct 31 02:18:15.872776 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.872734    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.872891 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.872867    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbfd\" (UniqueName: \"kubernetes.io/projected/8ed586a7-b174-4a71-9802-24e4880802d8-kube-api-access-hvbfd\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.872981 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.872965    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.973977 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.973940    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbfd\" (UniqueName: \"kubernetes.io/projected/8ed586a7-b174-4a71-9802-24e4880802d8-kube-api-access-hvbfd\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.975160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.975137    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.975320 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.975303    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.976421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.976385    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.981135 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.980987    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:15.988151 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:15.988125    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbfd\" (UniqueName: \"kubernetes.io/projected/8ed586a7-b174-4a71-9802-24e4880802d8-kube-api-access-hvbfd\") pod \"maas-api-revoke-sub-69a9ade1-89jmz\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:16.172845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:16.172810    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:16.357133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:16.356218    5293 generic.go:356] "Generic (PLEG): container finished" podID="283c48f8-8987-458f-8d6e-07ea7ab01754" containerID="dd8fc8ffebe3b1229ff240ab574ef5ca6d96ba772406b5d3a08f7f4ebffd32b6" exitCode=0
Oct 31 02:18:16.357133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:16.356381    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx" event={"ID":"283c48f8-8987-458f-8d6e-07ea7ab01754","Type":"ContainerDied","Data":"dd8fc8ffebe3b1229ff240ab574ef5ca6d96ba772406b5d3a08f7f4ebffd32b6"}
Oct 31 02:18:16.601739 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:16.601704    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"]
Oct 31 02:18:16.602496 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:18:16.602465    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed586a7_b174_4a71_9802_24e4880802d8.slice/crio-048a81ae2128af716faeecbc54d0075bd9261b72c32f5719cee6d9d6f16dc2cb WatchSource:0}: Error finding container 048a81ae2128af716faeecbc54d0075bd9261b72c32f5719cee6d9d6f16dc2cb: Status 404 returned error can't find the container with id 048a81ae2128af716faeecbc54d0075bd9261b72c32f5719cee6d9d6f16dc2cb
Oct 31 02:18:17.366854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.366765    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz" event={"ID":"8ed586a7-b174-4a71-9802-24e4880802d8","Type":"ContainerStarted","Data":"b8ac7b1a450e3c8f708a85b4c64dac6b791cbde80c23eb39bc5932ccda33b17b"}
Oct 31 02:18:17.366854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.366825    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz" event={"ID":"8ed586a7-b174-4a71-9802-24e4880802d8","Type":"ContainerStarted","Data":"048a81ae2128af716faeecbc54d0075bd9261b72c32f5719cee6d9d6f16dc2cb"}
Oct 31 02:18:17.389862 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.389798    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz" podStartSLOduration=2.38978187 podStartE2EDuration="2.38978187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:18:17.38575132 +0000 UTC m=+3455.334399719" watchObservedRunningTime="2026-10-31 02:18:17.38978187 +0000 UTC m=+3455.338430258"
Oct 31 02:18:17.438357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.438321    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7" podUID="f22ea351-d8b6-499f-98ed-068aa33808d0" containerName="payload-processing" containerID="cri-o://a22ec7223afc8e7954a9b60b6c80949e5d7aee8c041455fc7a54c3da028cbb93" gracePeriod=25
Oct 31 02:18:17.451339 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.451291    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw" podUID="ee13e9ec-c271-4076-9849-a843994efad4" containerName="payload-pre-processing" containerID="cri-o://1b4832e30e281477cec70d5ec984ec191ce57caa05a2799043513a97d8fe8356" gracePeriod=25
Oct 31 02:18:17.592500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.592469    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:17.601728 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.601702    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-openshift-ca\") pod \"283c48f8-8987-458f-8d6e-07ea7ab01754\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") "
Oct 31 02:18:17.601838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.601788    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-xks-ca\") pod \"283c48f8-8987-458f-8d6e-07ea7ab01754\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") "
Oct 31 02:18:17.601950 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.601934    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283c48f8-8987-458f-8d6e-07ea7ab01754-kube-api-access-c864b\" (UniqueName: \"kubernetes.io/projected/283c48f8-8987-458f-8d6e-07ea7ab01754-kube-api-access-c864b\") pod \"283c48f8-8987-458f-8d6e-07ea7ab01754\" (UID: \"283c48f8-8987-458f-8d6e-07ea7ab01754\") "
Oct 31 02:18:17.602131 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.602109    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-openshift-ca" pod "283c48f8-8987-458f-8d6e-07ea7ab01754" (UID: "283c48f8-8987-458f-8d6e-07ea7ab01754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:17.602440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.602411    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:17.604110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.604083    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-xks-ca" pod "283c48f8-8987-458f-8d6e-07ea7ab01754" (UID: "283c48f8-8987-458f-8d6e-07ea7ab01754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:17.604416 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.604395    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283c48f8-8987-458f-8d6e-07ea7ab01754-kube-api-access-c864b" pod "283c48f8-8987-458f-8d6e-07ea7ab01754" (UID: "283c48f8-8987-458f-8d6e-07ea7ab01754"). InnerVolumeSpecName "kube-api-access-c864b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:17.703640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.703602    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283c48f8-8987-458f-8d6e-07ea7ab01754-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:17.703640 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:17.703634    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c864b\" (UniqueName: \"kubernetes.io/projected/283c48f8-8987-458f-8d6e-07ea7ab01754-kube-api-access-c864b\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.023940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.023853    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"]
Oct 31 02:18:18.027095 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.027066    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda03711-qdq82"]
Oct 31 02:18:18.174845 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.174813    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7682f96-546f-4ef2-8c8e-ef2ce30b34b3" path="/var/lib/kubelet/pods/b7682f96-546f-4ef2-8c8e-ef2ce30b34b3/volumes"
Oct 31 02:18:18.387755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.387672    5293 generic.go:356] "Generic (PLEG): container finished" podID="8ed586a7-b174-4a71-9802-24e4880802d8" containerID="b8ac7b1a450e3c8f708a85b4c64dac6b791cbde80c23eb39bc5932ccda33b17b" exitCode=0
Oct 31 02:18:18.388157 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.387756    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz" event={"ID":"8ed586a7-b174-4a71-9802-24e4880802d8","Type":"ContainerDied","Data":"b8ac7b1a450e3c8f708a85b4c64dac6b791cbde80c23eb39bc5932ccda33b17b"}
Oct 31 02:18:18.390390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.390266    5293 generic.go:356] "Generic (PLEG): container finished" podID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerID="02bfbf1726a2eecb1b335896818cdfc46c1d8e5091113a135575ef92fc0da4fa" exitCode=0
Oct 31 02:18:18.390540 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.390337    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" event={"ID":"d5f02e10-5591-47c3-be44-5f9d7a3fc78f","Type":"ContainerDied","Data":"02bfbf1726a2eecb1b335896818cdfc46c1d8e5091113a135575ef92fc0da4fa"}
Oct 31 02:18:18.392411 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.392387    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx" event={"ID":"283c48f8-8987-458f-8d6e-07ea7ab01754","Type":"ContainerDied","Data":"c2cbe9c76d9afef8457a76ba5b7f10d118e0c3b000680e90096cb522d2070fa2"}
Oct 31 02:18:18.392517 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.392415    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"
Oct 31 02:18:18.392517 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.392432    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2cbe9c76d9afef8457a76ba5b7f10d118e0c3b000680e90096cb522d2070fa2"
Oct 31 02:18:18.444071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.443265    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"]
Oct 31 02:18:18.448221 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.448191    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae5a1c-ncxrt"]
Oct 31 02:18:18.493025 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.492998    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:18:18.524302 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524263    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-podinfo\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524480 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524310    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-socket\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524480 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524380    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istiod-ca-cert\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524480 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524453    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-data\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524501    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-credential-socket\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524568    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-certs\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524616    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-kube-api-access-l9nt4\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-kube-api-access-l9nt4\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524840 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524683    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-envoy\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524840 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524752    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-token\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-token\") pod \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\" (UID: \"d5f02e10-5591-47c3-be44-5f9d7a3fc78f\") "
Oct 31 02:18:18.524946 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.524863    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-data" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:18.525071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.525031    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-socket" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:18.525242 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.525166    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.525573 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.525554    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istiod-ca-cert" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:18.525669 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.525601    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-credential-socket" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:18.526569 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.526545    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-certs" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:18.528288 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.528265    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-kube-api-access-l9nt4" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "kube-api-access-l9nt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:18.528492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.528468    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-podinfo" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:18.528579 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.528485    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-envoy" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:18.530068 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.530027    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-token" pod "d5f02e10-5591-47c3-be44-5f9d7a3fc78f" (UID: "d5f02e10-5591-47c3-be44-5f9d7a3fc78f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:18.625941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.625907    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.625941 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.625940    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.626191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.625950    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9nt4\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-kube-api-access-l9nt4\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.626191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.625963    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.626191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.625975    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.626191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.625983    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.626191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.625992    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:18.626191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:18.626005    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f02e10-5591-47c3-be44-5f9d7a3fc78f-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:19.400526 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.400493    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"
Oct 31 02:18:19.400932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.400501    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5" event={"ID":"d5f02e10-5591-47c3-be44-5f9d7a3fc78f","Type":"ContainerDied","Data":"648a11f45a031b7c385ff812c30d41c90b6a7a0d2837a37be29ee2816bcf0c89"}
Oct 31 02:18:19.400932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.400636    5293 scope.go:122] "RemoveContainer" containerID="02bfbf1726a2eecb1b335896818cdfc46c1d8e5091113a135575ef92fc0da4fa"
Oct 31 02:18:19.429443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.429013    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"]
Oct 31 02:18:19.444159 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.444130    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-9kkc5"]
Oct 31 02:18:19.565546 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.565522    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:19.664585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.664503    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-xks-ca\") pod \"8ed586a7-b174-4a71-9802-24e4880802d8\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") "
Oct 31 02:18:19.664585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.664547    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-openshift-ca\") pod \"8ed586a7-b174-4a71-9802-24e4880802d8\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") "
Oct 31 02:18:19.664837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.664626    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed586a7-b174-4a71-9802-24e4880802d8-kube-api-access-hvbfd\" (UniqueName: \"kubernetes.io/projected/8ed586a7-b174-4a71-9802-24e4880802d8-kube-api-access-hvbfd\") pod \"8ed586a7-b174-4a71-9802-24e4880802d8\" (UID: \"8ed586a7-b174-4a71-9802-24e4880802d8\") "
Oct 31 02:18:19.665108 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.665034    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-openshift-ca" pod "8ed586a7-b174-4a71-9802-24e4880802d8" (UID: "8ed586a7-b174-4a71-9802-24e4880802d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:19.667017 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.666994    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-xks-ca" pod "8ed586a7-b174-4a71-9802-24e4880802d8" (UID: "8ed586a7-b174-4a71-9802-24e4880802d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:19.667167 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.667151    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed586a7-b174-4a71-9802-24e4880802d8-kube-api-access-hvbfd" pod "8ed586a7-b174-4a71-9802-24e4880802d8" (UID: "8ed586a7-b174-4a71-9802-24e4880802d8"). InnerVolumeSpecName "kube-api-access-hvbfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:19.766682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.766654    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:19.766682 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.766682    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed586a7-b174-4a71-9802-24e4880802d8-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:19.766916 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:19.766693    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvbfd\" (UniqueName: \"kubernetes.io/projected/8ed586a7-b174-4a71-9802-24e4880802d8-kube-api-access-hvbfd\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:20.173315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:20.173271    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" path="/var/lib/kubelet/pods/d5f02e10-5591-47c3-be44-5f9d7a3fc78f/volumes"
Oct 31 02:18:20.173760 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:20.173741    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59066d6-007e-42f2-b6a0-0f8c7e5b2403" path="/var/lib/kubelet/pods/e59066d6-007e-42f2-b6a0-0f8c7e5b2403/volumes"
Oct 31 02:18:20.410001 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:20.409969    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz" event={"ID":"8ed586a7-b174-4a71-9802-24e4880802d8","Type":"ContainerDied","Data":"048a81ae2128af716faeecbc54d0075bd9261b72c32f5719cee6d9d6f16dc2cb"}
Oct 31 02:18:20.410429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:20.410016    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="048a81ae2128af716faeecbc54d0075bd9261b72c32f5719cee6d9d6f16dc2cb"
Oct 31 02:18:20.410429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:20.410025    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69a9ade1-89jmz"
Oct 31 02:18:22.111276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.111241    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" containerID="cri-o://c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7" gracePeriod=45
Oct 31 02:18:22.114632 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.114601    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" containerID="cri-o://1d082e476327131484ed2f7462af756e23317fbf0eb4b3c4deafbd50e8f28c5d" gracePeriod=45
Oct 31 02:18:22.118474 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.118445    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main" containerID="cri-o://2922014f26b988552436afdce7ef37a5606c2cf1d9828c4eba7a0cff655ff032" gracePeriod=45
Oct 31 02:18:22.140999 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.140968    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7" podUID="f22ea351-d8b6-499f-98ed-068aa33808d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:22.141097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.141005    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw" podUID="ee13e9ec-c271-4076-9849-a843994efad4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:22.171903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.171874    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": dial tcp 10.133.0.70:8000: connect: connection refused"
Oct 31 02:18:22.273586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.273540    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused"
Oct 31 02:18:22.431647 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.431621    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:18:22.437137 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.437113    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:18:22.437788 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.437764    5293 generic.go:356] "Generic (PLEG): container finished" podID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerID="c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7" exitCode=0
Oct 31 02:18:22.437917 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.437804    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"
Oct 31 02:18:22.437917 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.437843    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" event={"ID":"4d52d8a0-8ca2-4d61-91f1-e992abdbe884","Type":"ContainerDied","Data":"c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7"}
Oct 31 02:18:22.437917 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.437884    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq" event={"ID":"4d52d8a0-8ca2-4d61-91f1-e992abdbe884","Type":"ContainerDied","Data":"8e9449745eb56ebd6cac969f5ff24ee22f44feeae4e9ebf887a8332572365f7a"}
Oct 31 02:18:22.437917 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.437901    5293 scope.go:122] "RemoveContainer" containerID="c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7"
Oct 31 02:18:22.439508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.439485    5293 generic.go:356] "Generic (PLEG): container finished" podID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerID="2922014f26b988552436afdce7ef37a5606c2cf1d9828c4eba7a0cff655ff032" exitCode=0
Oct 31 02:18:22.439610 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.439534    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"
Oct 31 02:18:22.439610 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.439573    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2" event={"ID":"d8ef0574-059a-42a7-b771-a474c4ca214b","Type":"ContainerDied","Data":"2922014f26b988552436afdce7ef37a5606c2cf1d9828c4eba7a0cff655ff032"}
Oct 31 02:18:22.441419 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.441400    5293 generic.go:356] "Generic (PLEG): container finished" podID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerID="1d082e476327131484ed2f7462af756e23317fbf0eb4b3c4deafbd50e8f28c5d" exitCode=0
Oct 31 02:18:22.441514 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.441446    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" event={"ID":"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5","Type":"ContainerDied","Data":"1d082e476327131484ed2f7462af756e23317fbf0eb4b3c4deafbd50e8f28c5d"}
Oct 31 02:18:22.448422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.448403    5293 scope.go:122] "RemoveContainer" containerID="c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7"
Oct 31 02:18:22.448704 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:22.448677    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7\": container with ID starting with c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7 not found: ID does not exist" containerID="c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7"
Oct 31 02:18:22.448803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.448715    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7"} err="failed to get container status \"c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7\": rpc error: code = NotFound desc = could not find container \"c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7\": container with ID starting with c692f964c0cd64cc1411337e2fbeeacfd9a58153ea757ca3656be2b5c4bd04a7 not found: ID does not exist"
Oct 31 02:18:22.448803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.448735    5293 scope.go:122] "RemoveContainer" containerID="2922014f26b988552436afdce7ef37a5606c2cf1d9828c4eba7a0cff655ff032"
Oct 31 02:18:22.494738 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.494715    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:18:22.504877 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.504836    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tls-certs\") pod \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") "
Oct 31 02:18:22.505050 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.504885    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-home\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-home\") pod \"d8ef0574-059a-42a7-b771-a474c4ca214b\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") "
Oct 31 02:18:22.505301 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.504968    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-tmp-dir\") pod \"d8ef0574-059a-42a7-b771-a474c4ca214b\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") "
Oct 31 02:18:22.505807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.505207    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-home" pod "d8ef0574-059a-42a7-b771-a474c4ca214b" (UID: "d8ef0574-059a-42a7-b771-a474c4ca214b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.505807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.505269    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-tmp-dir" pod "d8ef0574-059a-42a7-b771-a474c4ca214b" (UID: "d8ef0574-059a-42a7-b771-a474c4ca214b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.505807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.505437    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-model-cache\") pod \"d8ef0574-059a-42a7-b771-a474c4ca214b\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") "
Oct 31 02:18:22.505807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.505644    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-model-cache" pod "d8ef0574-059a-42a7-b771-a474c4ca214b" (UID: "d8ef0574-059a-42a7-b771-a474c4ca214b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.505807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.505718    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8ef0574-059a-42a7-b771-a474c4ca214b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ef0574-059a-42a7-b771-a474c4ca214b-tls-certs\") pod \"d8ef0574-059a-42a7-b771-a474c4ca214b\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") "
Oct 31 02:18:22.505807 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.505745    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ef0574-059a-42a7-b771-a474c4ca214b-kube-api-access-5s7ms\" (UniqueName: \"kubernetes.io/projected/d8ef0574-059a-42a7-b771-a474c4ca214b-kube-api-access-5s7ms\") pod \"d8ef0574-059a-42a7-b771-a474c4ca214b\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") "
Oct 31 02:18:22.506191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.506074    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-model-cache\") pod \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") "
Oct 31 02:18:22.506191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.506136    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tmp-dir\") pod \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") "
Oct 31 02:18:22.506191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.506171    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-dshm\") pod \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") "
Oct 31 02:18:22.506354 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.506242    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-kube-api-access-mhq4x\" (UniqueName: \"kubernetes.io/projected/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-kube-api-access-mhq4x\") pod \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") "
Oct 31 02:18:22.506412 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.506381    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-home\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-home\") pod \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\" (UID: \"4d52d8a0-8ca2-4d61-91f1-e992abdbe884\") "
Oct 31 02:18:22.506592 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.506573    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-dshm\") pod \"d8ef0574-059a-42a7-b771-a474c4ca214b\" (UID: \"d8ef0574-059a-42a7-b771-a474c4ca214b\") "
Oct 31 02:18:22.506723 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.506679    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tmp-dir" pod "4d52d8a0-8ca2-4d61-91f1-e992abdbe884" (UID: "4d52d8a0-8ca2-4d61-91f1-e992abdbe884"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.507207 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.507167    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-home" pod "4d52d8a0-8ca2-4d61-91f1-e992abdbe884" (UID: "4d52d8a0-8ca2-4d61-91f1-e992abdbe884"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.508362 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.508343    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.508548 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.508530    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.508655 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.508551    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.508655 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.508564    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.508655 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.508575    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.508655 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.508493    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-model-cache" pod "4d52d8a0-8ca2-4d61-91f1-e992abdbe884" (UID: "4d52d8a0-8ca2-4d61-91f1-e992abdbe884"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.519632 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.519577    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-dshm" pod "d8ef0574-059a-42a7-b771-a474c4ca214b" (UID: "d8ef0574-059a-42a7-b771-a474c4ca214b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.522085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.522031    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ef0574-059a-42a7-b771-a474c4ca214b-kube-api-access-5s7ms" pod "d8ef0574-059a-42a7-b771-a474c4ca214b" (UID: "d8ef0574-059a-42a7-b771-a474c4ca214b"). InnerVolumeSpecName "kube-api-access-5s7ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:22.522085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.522152    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ef0574-059a-42a7-b771-a474c4ca214b-tls-certs" pod "d8ef0574-059a-42a7-b771-a474c4ca214b" (UID: "d8ef0574-059a-42a7-b771-a474c4ca214b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:22.522085 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.522242    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tls-certs" pod "4d52d8a0-8ca2-4d61-91f1-e992abdbe884" (UID: "4d52d8a0-8ca2-4d61-91f1-e992abdbe884"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:22.523119 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.523083    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-kube-api-access-mhq4x" pod "4d52d8a0-8ca2-4d61-91f1-e992abdbe884" (UID: "4d52d8a0-8ca2-4d61-91f1-e992abdbe884"). InnerVolumeSpecName "kube-api-access-mhq4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:22.523762 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.523733    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-dshm" pod "4d52d8a0-8ca2-4d61-91f1-e992abdbe884" (UID: "4d52d8a0-8ca2-4d61-91f1-e992abdbe884"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.609907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.609878    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tmp-dir\") pod \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") "
Oct 31 02:18:22.610110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.609997    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-model-cache\") pod \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") "
Oct 31 02:18:22.610110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610025    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-home\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-home\") pod \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") "
Oct 31 02:18:22.610110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610053    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tmp-dir" pod "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" (UID: "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.610110 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610106    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-dshm\") pod \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") "
Oct 31 02:18:22.610451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610147    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-kube-api-access-wzj8j\" (UniqueName: \"kubernetes.io/projected/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-kube-api-access-wzj8j\") pod \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") "
Oct 31 02:18:22.610451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610202    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tls-certs\") pod \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\" (UID: \"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5\") "
Oct 31 02:18:22.610451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610211    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-model-cache" pod "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" (UID: "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.610451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610380    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-home" pod "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" (UID: "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.610765 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610747    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhq4x\" (UniqueName: \"kubernetes.io/projected/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-kube-api-access-mhq4x\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.610838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610770    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.610838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610785    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ef0574-059a-42a7-b771-a474c4ca214b-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.610838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610798    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.610838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610809    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.610838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610817    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.610838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610825    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ef0574-059a-42a7-b771-a474c4ca214b-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.610838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610833    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7ms\" (UniqueName: \"kubernetes.io/projected/d8ef0574-059a-42a7-b771-a474c4ca214b-kube-api-access-5s7ms\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.610838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610841    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.611234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.610850    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d52d8a0-8ca2-4d61-91f1-e992abdbe884-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.612819 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.612798    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-kube-api-access-wzj8j" pod "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" (UID: "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5"). InnerVolumeSpecName "kube-api-access-wzj8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:22.612924 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.612862    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-dshm" pod "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" (UID: "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.612924 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.612868    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tls-certs" pod "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" (UID: "e2a8cc67-aab8-4fc2-837e-97089ee9e3d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:22.712052 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.712007    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.712052 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.712049    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzj8j\" (UniqueName: \"kubernetes.io/projected/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-kube-api-access-wzj8j\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.712251 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.712064    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.761451 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.761422    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"]
Oct 31 02:18:22.765443 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.765415    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-w1-0e8b5c-kserve-7dcd48c865-ntdgq"]
Oct 31 02:18:22.776097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.776074    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"]
Oct 31 02:18:22.782022 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:22.781997    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/premium-w1-0e8b5c-kserve-7f895d78f8-lc8r2"]
Oct 31 02:18:23.452013 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:23.451979    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"
Oct 31 02:18:23.452013 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:23.451997    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4" event={"ID":"e2a8cc67-aab8-4fc2-837e-97089ee9e3d5","Type":"ContainerDied","Data":"c995425c372f140a788bf17a9676087484706ec29c26aeaf5da15076a4b1f59d"}
Oct 31 02:18:23.452618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:23.452100    5293 scope.go:122] "RemoveContainer" containerID="1d082e476327131484ed2f7462af756e23317fbf0eb4b3c4deafbd50e8f28c5d"
Oct 31 02:18:23.476390 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:23.476366    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"]
Oct 31 02:18:23.480739 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:23.480711    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/distinct-2-w1-0e8b5c-kserve-7bc84c646b-s9gc4"]
Oct 31 02:18:24.094163 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.094131    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"]
Oct 31 02:18:24.109817 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.109774    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"]
Oct 31 02:18:24.173529 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.173499    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" path="/var/lib/kubelet/pods/4d52d8a0-8ca2-4d61-91f1-e992abdbe884/volumes"
Oct 31 02:18:24.173955 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.173932    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" path="/var/lib/kubelet/pods/d8ef0574-059a-42a7-b771-a474c4ca214b/volumes"
Oct 31 02:18:24.174466 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.174446    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" path="/var/lib/kubelet/pods/e2a8cc67-aab8-4fc2-837e-97089ee9e3d5/volumes"
Oct 31 02:18:24.622284 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622251    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"]
Oct 31 02:18:24.622714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622700    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy"
Oct 31 02:18:24.622714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622714    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy"
Oct 31 02:18:24.622791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622731    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8ed586a7-b174-4a71-9802-24e4880802d8" containerName="revoke-keys"
Oct 31 02:18:24.622791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622736    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed586a7-b174-4a71-9802-24e4880802d8" containerName="revoke-keys"
Oct 31 02:18:24.622791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622761    5293 cpu_manager.go:414] "containerMap: removing container" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main"
Oct 31 02:18:24.622791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622768    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main"
Oct 31 02:18:24.622791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622780    5293 cpu_manager.go:414] "containerMap: removing container" podUID="283c48f8-8987-458f-8d6e-07ea7ab01754" containerName="revoke-keys"
Oct 31 02:18:24.622791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622785    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283c48f8-8987-458f-8d6e-07ea7ab01754" containerName="revoke-keys"
Oct 31 02:18:24.622791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622791    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622795    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622804    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622808    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622871    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d52d8a0-8ca2-4d61-91f1-e992abdbe884" containerName="main"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622879    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="283c48f8-8987-458f-8d6e-07ea7ab01754" containerName="revoke-keys"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622885    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a8cc67-aab8-4fc2-837e-97089ee9e3d5" containerName="main"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622893    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ef0574-059a-42a7-b771-a474c4ca214b" containerName="main"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622898    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f02e10-5591-47c3-be44-5f9d7a3fc78f" containerName="istio-proxy"
Oct 31 02:18:24.623000 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.622904    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed586a7-b174-4a71-9802-24e4880802d8" containerName="revoke-keys"
Oct 31 02:18:24.628494 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.628471    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"]
Oct 31 02:18:24.628618 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.628596    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.632212 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.632189    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:18:24.632712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.632523    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.632712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.632537    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:18:24.637234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.637213    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"]
Oct 31 02:18:24.644136 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.644112    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"]
Oct 31 02:18:24.661071 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.661023    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:18:24.661186 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.661154    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:18:24.748869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.748839    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512bffc0-b64c-461c-a374-4530f745567f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.749018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.748873    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512bffc0-b64c-461c-a374-4530f745567f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.749018 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.748910    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qj7v\" (UniqueName: \"kubernetes.io/projected/512bffc0-b64c-461c-a374-4530f745567f-kube-api-access-6qj7v\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.749134 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.749061    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbps\" (UniqueName: \"kubernetes.io/projected/e72d05ac-fcfe-450a-9656-772d37af4b65-kube-api-access-5tbps\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.749174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.749137    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.749211 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.749177    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.850566 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.850535    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.850729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.850636    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512bffc0-b64c-461c-a374-4530f745567f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.850729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.850656    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512bffc0-b64c-461c-a374-4530f745567f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.850729 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.850688    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qj7v\" (UniqueName: \"kubernetes.io/projected/512bffc0-b64c-461c-a374-4530f745567f-kube-api-access-6qj7v\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.850907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.850771    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbps\" (UniqueName: \"kubernetes.io/projected/e72d05ac-fcfe-450a-9656-772d37af4b65-kube-api-access-5tbps\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.850907 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.850819    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.851386 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.851351    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.851520 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.851420    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512bffc0-b64c-461c-a374-4530f745567f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.853244 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.853220    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.853375 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.853277    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512bffc0-b64c-461c-a374-4530f745567f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.862428 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.862408    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qj7v\" (UniqueName: \"kubernetes.io/projected/512bffc0-b64c-461c-a374-4530f745567f-kube-api-access-6qj7v\") pod \"maas-api-revoke-sub-5328a1f0-cftbg\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.862554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.862408    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbps\" (UniqueName: \"kubernetes.io/projected/e72d05ac-fcfe-450a-9656-772d37af4b65-kube-api-access-5tbps\") pod \"maas-api-revoke-sub-fca7cb37-nvzcr\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:24.957796 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.955700    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:24.963180 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:24.963146    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:25.252289 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:25.252262    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"]
Oct 31 02:18:25.253137 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:18:25.253109    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512bffc0_b64c_461c_a374_4530f745567f.slice/crio-3daa7db2f9ea4e7ae551071e833284dbf8144b965a9929e0825e047571137b69 WatchSource:0}: Error finding container 3daa7db2f9ea4e7ae551071e833284dbf8144b965a9929e0825e047571137b69: Status 404 returned error can't find the container with id 3daa7db2f9ea4e7ae551071e833284dbf8144b965a9929e0825e047571137b69
Oct 31 02:18:25.255885 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:25.255777    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"]
Oct 31 02:18:25.257635 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:18:25.257611    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72d05ac_fcfe_450a_9656_772d37af4b65.slice/crio-c6ab8d84891789c4342ae54f5cd72583e999f8a07e12fbd554ab283181c19689 WatchSource:0}: Error finding container c6ab8d84891789c4342ae54f5cd72583e999f8a07e12fbd554ab283181c19689: Status 404 returned error can't find the container with id c6ab8d84891789c4342ae54f5cd72583e999f8a07e12fbd554ab283181c19689
Oct 31 02:18:25.477933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:25.477892    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr" event={"ID":"e72d05ac-fcfe-450a-9656-772d37af4b65","Type":"ContainerStarted","Data":"c6ab8d84891789c4342ae54f5cd72583e999f8a07e12fbd554ab283181c19689"}
Oct 31 02:18:25.479425 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:25.479390    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg" event={"ID":"512bffc0-b64c-461c-a374-4530f745567f","Type":"ContainerStarted","Data":"3daa7db2f9ea4e7ae551071e833284dbf8144b965a9929e0825e047571137b69"}
Oct 31 02:18:25.833219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:25.833131    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"]
Oct 31 02:18:25.833633 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:25.833368    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" containerID="cri-o://27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6" gracePeriod=30
Oct 31 02:18:26.426399 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:26.426364    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"]
Oct 31 02:18:26.439079 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:26.439025    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e8b5c-5g556"]
Oct 31 02:18:26.498315 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:26.498281    5293 generic.go:356] "Generic (PLEG): container finished" podID="512bffc0-b64c-461c-a374-4530f745567f" containerID="8aca173a30527264fe8517ca40469a4800da7e1692cf893f73e427d0d0c68dc4" exitCode=0
Oct 31 02:18:26.498596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:26.498575    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg" event={"ID":"512bffc0-b64c-461c-a374-4530f745567f","Type":"ContainerDied","Data":"8aca173a30527264fe8517ca40469a4800da7e1692cf893f73e427d0d0c68dc4"}
Oct 31 02:18:26.501571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:26.501506    5293 generic.go:356] "Generic (PLEG): container finished" podID="e72d05ac-fcfe-450a-9656-772d37af4b65" containerID="64c24a5caf6f0522e61e45bdbc805755a2d10d07e999fa7661b79c98ba0ca1d0" exitCode=0
Oct 31 02:18:26.501571 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:26.501546    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr" event={"ID":"e72d05ac-fcfe-450a-9656-772d37af4b65","Type":"ContainerDied","Data":"64c24a5caf6f0522e61e45bdbc805755a2d10d07e999fa7661b79c98ba0ca1d0"}
Oct 31 02:18:27.672986 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.672960    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:27.701604 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.701575    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:27.783153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.783059    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512bffc0-b64c-461c-a374-4530f745567f-kube-api-access-6qj7v\" (UniqueName: \"kubernetes.io/projected/512bffc0-b64c-461c-a374-4530f745567f-kube-api-access-6qj7v\") pod \"512bffc0-b64c-461c-a374-4530f745567f\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") "
Oct 31 02:18:27.783153 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.783113    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/512bffc0-b64c-461c-a374-4530f745567f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512bffc0-b64c-461c-a374-4530f745567f-maas-api-xks-ca\") pod \"512bffc0-b64c-461c-a374-4530f745567f\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") "
Oct 31 02:18:27.783397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.783160    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512bffc0-b64c-461c-a374-4530f745567f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512bffc0-b64c-461c-a374-4530f745567f-maas-api-openshift-ca\") pod \"512bffc0-b64c-461c-a374-4530f745567f\" (UID: \"512bffc0-b64c-461c-a374-4530f745567f\") "
Oct 31 02:18:27.783397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.783244    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-openshift-ca\") pod \"e72d05ac-fcfe-450a-9656-772d37af4b65\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") "
Oct 31 02:18:27.783397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.783295    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72d05ac-fcfe-450a-9656-772d37af4b65-kube-api-access-5tbps\" (UniqueName: \"kubernetes.io/projected/e72d05ac-fcfe-450a-9656-772d37af4b65-kube-api-access-5tbps\") pod \"e72d05ac-fcfe-450a-9656-772d37af4b65\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") "
Oct 31 02:18:27.783397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.783365    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-xks-ca\") pod \"e72d05ac-fcfe-450a-9656-772d37af4b65\" (UID: \"e72d05ac-fcfe-450a-9656-772d37af4b65\") "
Oct 31 02:18:27.783639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.783593    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512bffc0-b64c-461c-a374-4530f745567f-maas-api-openshift-ca" pod "512bffc0-b64c-461c-a374-4530f745567f" (UID: "512bffc0-b64c-461c-a374-4530f745567f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:27.783960 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.783929    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-openshift-ca" pod "e72d05ac-fcfe-450a-9656-772d37af4b65" (UID: "e72d05ac-fcfe-450a-9656-772d37af4b65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:27.784197 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.784115    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512bffc0-b64c-461c-a374-4530f745567f-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:27.786324 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.786288    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-xks-ca" pod "e72d05ac-fcfe-450a-9656-772d37af4b65" (UID: "e72d05ac-fcfe-450a-9656-772d37af4b65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:27.786425 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.786297    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72d05ac-fcfe-450a-9656-772d37af4b65-kube-api-access-5tbps" pod "e72d05ac-fcfe-450a-9656-772d37af4b65" (UID: "e72d05ac-fcfe-450a-9656-772d37af4b65"). InnerVolumeSpecName "kube-api-access-5tbps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:27.791843 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.791820    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512bffc0-b64c-461c-a374-4530f745567f-kube-api-access-6qj7v" pod "512bffc0-b64c-461c-a374-4530f745567f" (UID: "512bffc0-b64c-461c-a374-4530f745567f"). InnerVolumeSpecName "kube-api-access-6qj7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:27.793260 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.793240    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512bffc0-b64c-461c-a374-4530f745567f-maas-api-xks-ca" pod "512bffc0-b64c-461c-a374-4530f745567f" (UID: "512bffc0-b64c-461c-a374-4530f745567f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:27.884932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.884897    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbps\" (UniqueName: \"kubernetes.io/projected/e72d05ac-fcfe-450a-9656-772d37af4b65-kube-api-access-5tbps\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:27.884932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.884927    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:27.884932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.884938    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qj7v\" (UniqueName: \"kubernetes.io/projected/512bffc0-b64c-461c-a374-4530f745567f-kube-api-access-6qj7v\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:27.885205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.884948    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512bffc0-b64c-461c-a374-4530f745567f-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:27.885205 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:27.884957    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72d05ac-fcfe-450a-9656-772d37af4b65-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:28.025638 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.025610    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"]
Oct 31 02:18:28.028865 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.028839    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfb99a85-vw5z2"]
Oct 31 02:18:28.174115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.174080    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5163832d-414a-424c-8c48-0ba095d76b12" path="/var/lib/kubelet/pods/5163832d-414a-424c-8c48-0ba095d76b12/volumes"
Oct 31 02:18:28.174730 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.174713    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa" path="/var/lib/kubelet/pods/f7edbdfd-be8a-4f08-a2a0-8ed8b84547fa/volumes"
Oct 31 02:18:28.517795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.517704    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr"
Oct 31 02:18:28.517795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.517725    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca7cb37-nvzcr" event={"ID":"e72d05ac-fcfe-450a-9656-772d37af4b65","Type":"ContainerDied","Data":"c6ab8d84891789c4342ae54f5cd72583e999f8a07e12fbd554ab283181c19689"}
Oct 31 02:18:28.517795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.517786    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ab8d84891789c4342ae54f5cd72583e999f8a07e12fbd554ab283181c19689"
Oct 31 02:18:28.519397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.519375    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg"
Oct 31 02:18:28.519575 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.519410    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a1f0-cftbg" event={"ID":"512bffc0-b64c-461c-a374-4530f745567f","Type":"ContainerDied","Data":"3daa7db2f9ea4e7ae551071e833284dbf8144b965a9929e0825e047571137b69"}
Oct 31 02:18:28.519575 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:28.519448    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3daa7db2f9ea4e7ae551071e833284dbf8144b965a9929e0825e047571137b69"
Oct 31 02:18:29.094777 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:29.094728    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt" podUID="3cb196ab-0bf0-4a69-9b1f-61328e7a2804" containerName="payload-processing" containerID="cri-o://5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8" gracePeriod=25
Oct 31 02:18:29.110426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:29.110358    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h" podUID="082b7243-443f-49c1-90ad-955cbe31ec7f" containerName="payload-pre-processing" containerID="cri-o://b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2" gracePeriod=25
Oct 31 02:18:30.102938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:30.102903    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt" podUID="3cb196ab-0bf0-4a69-9b1f-61328e7a2804" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:30.103469 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:30.102900    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h" podUID="082b7243-443f-49c1-90ad-955cbe31ec7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:30.264624 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:30.264586    5293 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:18:30.264792 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:30.264659    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:18:31.107544 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.107485    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 02:18:31.224713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.224679    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istiod-ca-cert\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.224896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.224730    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-socket\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.224896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.224760    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-token\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-token\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.224896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.224842    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-podinfo\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.225128 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.224901    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-kube-api-access-4pwn7\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-kube-api-access-4pwn7\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.225128 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.224985    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-certs\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.225256 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225167    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-data\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.225256 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225180    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istiod-ca-cert" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:31.225256 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225201    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-credential-socket\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.225420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225299    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-envoy\") pod \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\" (UID: \"8baa7adb-dc5a-4f8c-b24a-ced615febf3b\") "
Oct 31 02:18:31.225420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225405    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-socket" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.225532 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225467    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-certs" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.225748 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225609    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-credential-socket" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.225748 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225635    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-data" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.225879 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225834    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.225879 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225855    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.225879 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225870    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.226081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225884    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.226081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.225898    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.228117 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.228082    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-podinfo" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:31.228251 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.228223    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-token" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:31.228392 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.228371    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-kube-api-access-4pwn7" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "kube-api-access-4pwn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:31.228439 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.228378    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-envoy" pod "8baa7adb-dc5a-4f8c-b24a-ced615febf3b" (UID: "8baa7adb-dc5a-4f8c-b24a-ced615febf3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.327375 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.327335    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.327375 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.327376    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.327597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.327391    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwn7\" (UniqueName: \"kubernetes.io/projected/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-kube-api-access-4pwn7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.327597 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.327407    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baa7adb-dc5a-4f8c-b24a-ced615febf3b-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.350192 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.350145    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"]
Oct 31 02:18:31.354875 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.354843    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"]
Oct 31 02:18:31.378711 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.378636    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"]
Oct 31 02:18:31.569346 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.569314    5293 generic.go:356] "Generic (PLEG): container finished" podID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerID="27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6" exitCode=0
Oct 31 02:18:31.569525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.569370    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"
Oct 31 02:18:31.569525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.569404    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" event={"ID":"8baa7adb-dc5a-4f8c-b24a-ced615febf3b","Type":"ContainerDied","Data":"27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6"}
Oct 31 02:18:31.569525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.569479    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2" event={"ID":"8baa7adb-dc5a-4f8c-b24a-ced615febf3b","Type":"ContainerDied","Data":"c63f71587b7e2f779dddf5a26ccbd1f25bab0eaaa836e1df42d10c9d712fa630"}
Oct 31 02:18:31.569525 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.569507    5293 scope.go:122] "RemoveContainer" containerID="27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6"
Oct 31 02:18:31.584735 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.584717    5293 scope.go:122] "RemoveContainer" containerID="27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6"
Oct 31 02:18:31.585082 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:31.585057    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6\": container with ID starting with 27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6 not found: ID does not exist" containerID="27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6"
Oct 31 02:18:31.585141 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.585097    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6"} err="failed to get container status \"27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6\": rpc error: code = NotFound desc = could not find container \"27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6\": container with ID starting with 27166f88559d2230e313441a052d0d84ace2f9a4bb8eb52f4c2d70f878eedbf6 not found: ID does not exist"
Oct 31 02:18:31.596894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.596867    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"]
Oct 31 02:18:31.602858 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.602837    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-dvkb2"]
Oct 31 02:18:31.811347 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:31.811308    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:18:32.141813 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:32.141720    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7" podUID="f22ea351-d8b6-499f-98ed-068aa33808d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:32.142247 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:32.141734    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw" podUID="ee13e9ec-c271-4076-9849-a843994efad4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:32.189426 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:32.189389    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" path="/var/lib/kubelet/pods/8baa7adb-dc5a-4f8c-b24a-ced615febf3b/volumes"
Oct 31 02:18:35.023056 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:35.023008    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"]
Oct 31 02:18:35.023994 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:35.023944    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" containerID="cri-o://e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391" gracePeriod=30
Oct 31 02:18:35.026538 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:35.026502    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"]
Oct 31 02:18:35.026712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:35.026689    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" containerID="cri-o://e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5" gracePeriod=30
Oct 31 02:18:35.336127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:35.336002    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:18:35.336127 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:35.336095    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:18:36.526814 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:36.526774    5293 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:18:36.527279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:36.526849    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:18:36.785317 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:36.785236    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"]
Oct 31 02:18:37.121114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121012    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"]
Oct 31 02:18:37.121546 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121531    5293 cpu_manager.go:414] "containerMap: removing container" podUID="512bffc0-b64c-461c-a374-4530f745567f" containerName="revoke-keys"
Oct 31 02:18:37.121593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121548    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512bffc0-b64c-461c-a374-4530f745567f" containerName="revoke-keys"
Oct 31 02:18:37.121593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121562    5293 cpu_manager.go:414] "containerMap: removing container" podUID="e72d05ac-fcfe-450a-9656-772d37af4b65" containerName="revoke-keys"
Oct 31 02:18:37.121593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121566    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72d05ac-fcfe-450a-9656-772d37af4b65" containerName="revoke-keys"
Oct 31 02:18:37.121593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121585    5293 cpu_manager.go:414] "containerMap: removing container" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy"
Oct 31 02:18:37.121593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121591    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy"
Oct 31 02:18:37.121793 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121668    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="8baa7adb-dc5a-4f8c-b24a-ced615febf3b" containerName="istio-proxy"
Oct 31 02:18:37.121793 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121681    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="512bffc0-b64c-461c-a374-4530f745567f" containerName="revoke-keys"
Oct 31 02:18:37.121793 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.121687    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72d05ac-fcfe-450a-9656-772d37af4b65" containerName="revoke-keys"
Oct 31 02:18:37.126434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.126412    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.131176 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.131151    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:18:37.131599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.131576    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:18:37.131791 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.131778    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:18:37.131885 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.131238    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:18:37.137097 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.137076    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"]
Oct 31 02:18:37.200363 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.200307    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5fq6\" (UniqueName: \"kubernetes.io/projected/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-kube-api-access-r5fq6\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.200563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.200440    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.200563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.200487    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.301909 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.301866    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5fq6\" (UniqueName: \"kubernetes.io/projected/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-kube-api-access-r5fq6\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.302271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.301917    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.302271 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.301953    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.302712 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.302689    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.304628 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.304601    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.314175 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.314152    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5fq6\" (UniqueName: \"kubernetes.io/projected/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-kube-api-access-r5fq6\") pod \"maas-api-revoke-sub-8fb9a6e2-hsf5t\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.459623 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.459598    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:37.621751 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.621526    5293 generic.go:356] "Generic (PLEG): container finished" podID="f22ea351-d8b6-499f-98ed-068aa33808d0" containerID="a22ec7223afc8e7954a9b60b6c80949e5d7aee8c041455fc7a54c3da028cbb93" exitCode=0
Oct 31 02:18:37.621751 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.621702    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7" event={"ID":"f22ea351-d8b6-499f-98ed-068aa33808d0","Type":"ContainerDied","Data":"a22ec7223afc8e7954a9b60b6c80949e5d7aee8c041455fc7a54c3da028cbb93"}
Oct 31 02:18:37.626081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.624697    5293 generic.go:356] "Generic (PLEG): container finished" podID="ee13e9ec-c271-4076-9849-a843994efad4" containerID="1b4832e30e281477cec70d5ec984ec191ce57caa05a2799043513a97d8fe8356" exitCode=0
Oct 31 02:18:37.626081 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.624820    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw" event={"ID":"ee13e9ec-c271-4076-9849-a843994efad4","Type":"ContainerDied","Data":"1b4832e30e281477cec70d5ec984ec191ce57caa05a2799043513a97d8fe8356"}
Oct 31 02:18:37.680446 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.680422    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 02:18:37.712506 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.712475    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 02:18:37.749962 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.749937    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"]
Oct 31 02:18:37.751448 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:18:37.751423    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1c02f9_0059_4a76_a9b2_ed09a9ffb830.slice/crio-e90a876abf054a5ee66ee3c4c70da7465bd9256b7f87b6b5545b3b0e398de603 WatchSource:0}: Error finding container e90a876abf054a5ee66ee3c4c70da7465bd9256b7f87b6b5545b3b0e398de603: Status 404 returned error can't find the container with id e90a876abf054a5ee66ee3c4c70da7465bd9256b7f87b6b5545b3b0e398de603
Oct 31 02:18:37.807014 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.806989    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee13e9ec-c271-4076-9849-a843994efad4-kube-api-access-s2fk6\" (UniqueName: \"kubernetes.io/projected/ee13e9ec-c271-4076-9849-a843994efad4-kube-api-access-s2fk6\") pod \"ee13e9ec-c271-4076-9849-a843994efad4\" (UID: \"ee13e9ec-c271-4076-9849-a843994efad4\") "
Oct 31 02:18:37.807165 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.807100    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee13e9ec-c271-4076-9849-a843994efad4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee13e9ec-c271-4076-9849-a843994efad4-plugins-config-volume\") pod \"ee13e9ec-c271-4076-9849-a843994efad4\" (UID: \"ee13e9ec-c271-4076-9849-a843994efad4\") "
Oct 31 02:18:37.807228 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.807208    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22ea351-d8b6-499f-98ed-068aa33808d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22ea351-d8b6-499f-98ed-068aa33808d0-plugins-config-volume\") pod \"f22ea351-d8b6-499f-98ed-068aa33808d0\" (UID: \"f22ea351-d8b6-499f-98ed-068aa33808d0\") "
Oct 31 02:18:37.807294 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.807245    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22ea351-d8b6-499f-98ed-068aa33808d0-kube-api-access-v2cw8\" (UniqueName: \"kubernetes.io/projected/f22ea351-d8b6-499f-98ed-068aa33808d0-kube-api-access-v2cw8\") pod \"f22ea351-d8b6-499f-98ed-068aa33808d0\" (UID: \"f22ea351-d8b6-499f-98ed-068aa33808d0\") "
Oct 31 02:18:37.807599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.807572    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee13e9ec-c271-4076-9849-a843994efad4-plugins-config-volume" pod "ee13e9ec-c271-4076-9849-a843994efad4" (UID: "ee13e9ec-c271-4076-9849-a843994efad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:37.807599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.807584    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22ea351-d8b6-499f-98ed-068aa33808d0-plugins-config-volume" pod "f22ea351-d8b6-499f-98ed-068aa33808d0" (UID: "f22ea351-d8b6-499f-98ed-068aa33808d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:37.809519 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.809495    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee13e9ec-c271-4076-9849-a843994efad4-kube-api-access-s2fk6" pod "ee13e9ec-c271-4076-9849-a843994efad4" (UID: "ee13e9ec-c271-4076-9849-a843994efad4"). InnerVolumeSpecName "kube-api-access-s2fk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:37.809639 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.809563    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22ea351-d8b6-499f-98ed-068aa33808d0-kube-api-access-v2cw8" pod "f22ea351-d8b6-499f-98ed-068aa33808d0" (UID: "f22ea351-d8b6-499f-98ed-068aa33808d0"). InnerVolumeSpecName "kube-api-access-v2cw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:37.908894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.908847    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2fk6\" (UniqueName: \"kubernetes.io/projected/ee13e9ec-c271-4076-9849-a843994efad4-kube-api-access-s2fk6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:37.908894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.908887    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee13e9ec-c271-4076-9849-a843994efad4-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:37.908894 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.908901    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22ea351-d8b6-499f-98ed-068aa33808d0-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:37.909150 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:37.908914    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2cw8\" (UniqueName: \"kubernetes.io/projected/f22ea351-d8b6-499f-98ed-068aa33808d0-kube-api-access-v2cw8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:38.633788 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.633760    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"
Oct 31 02:18:38.634254 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.633784    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw" event={"ID":"ee13e9ec-c271-4076-9849-a843994efad4","Type":"ContainerDied","Data":"b854f53ea89447668d22197ec132cddc57fdf3ff5a1bd84e86da40823e0a369a"}
Oct 31 02:18:38.634254 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.633860    5293 scope.go:122] "RemoveContainer" containerID="1b4832e30e281477cec70d5ec984ec191ce57caa05a2799043513a97d8fe8356"
Oct 31 02:18:38.636626 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.635882    5293 generic.go:356] "Generic (PLEG): container finished" podID="6b1c02f9-0059-4a76-a9b2-ed09a9ffb830" containerID="d206988c065b6e97f2802539b03ab76c3cd722b895a0bc9a4820de040dc40aa0" exitCode=0
Oct 31 02:18:38.636626 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.635984    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t" event={"ID":"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830","Type":"ContainerDied","Data":"d206988c065b6e97f2802539b03ab76c3cd722b895a0bc9a4820de040dc40aa0"}
Oct 31 02:18:38.636626 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.636030    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t" event={"ID":"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830","Type":"ContainerStarted","Data":"e90a876abf054a5ee66ee3c4c70da7465bd9256b7f87b6b5545b3b0e398de603"}
Oct 31 02:18:38.637916 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.637888    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"
Oct 31 02:18:38.638055 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.637937    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7" event={"ID":"f22ea351-d8b6-499f-98ed-068aa33808d0","Type":"ContainerDied","Data":"55474a2228d8c009871802bf24bccd4f9ac153a2145d7a464e3bcca821fb0e0e"}
Oct 31 02:18:38.645812 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.645787    5293 scope.go:122] "RemoveContainer" containerID="a22ec7223afc8e7954a9b60b6c80949e5d7aee8c041455fc7a54c3da028cbb93"
Oct 31 02:18:38.651952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.651932    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"]
Oct 31 02:18:38.656583 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.656564    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0e8b5c-55758f85c9-n7hhw"]
Oct 31 02:18:38.686008 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.685980    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"]
Oct 31 02:18:38.690340 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:38.690315    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e8b5c-5fdcffb6c5-rqzn7"]
Oct 31 02:18:39.345381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.345339    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"]
Oct 31 02:18:39.348427 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.348398    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6ae519-4qlcx"]
Oct 31 02:18:39.818585 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.818555    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:39.950856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.950753    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-xks-ca\") pod \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") "
Oct 31 02:18:39.950856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.950828    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-openshift-ca\") pod \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") "
Oct 31 02:18:39.951164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.951004    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-kube-api-access-r5fq6\" (UniqueName: \"kubernetes.io/projected/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-kube-api-access-r5fq6\") pod \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\" (UID: \"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830\") "
Oct 31 02:18:39.951570 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.951538    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-openshift-ca" pod "6b1c02f9-0059-4a76-a9b2-ed09a9ffb830" (UID: "6b1c02f9-0059-4a76-a9b2-ed09a9ffb830"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:39.954435 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.954402    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-kube-api-access-r5fq6" pod "6b1c02f9-0059-4a76-a9b2-ed09a9ffb830" (UID: "6b1c02f9-0059-4a76-a9b2-ed09a9ffb830"). InnerVolumeSpecName "kube-api-access-r5fq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:39.954435 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:39.954420    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-xks-ca" pod "6b1c02f9-0059-4a76-a9b2-ed09a9ffb830" (UID: "6b1c02f9-0059-4a76-a9b2-ed09a9ffb830"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:40.025309 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.025273    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"]
Oct 31 02:18:40.029576 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.029534    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a2835ee-24z6t"]
Oct 31 02:18:40.052753 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.052711    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5fq6\" (UniqueName: \"kubernetes.io/projected/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-kube-api-access-r5fq6\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.052753 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.052753    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-xks-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.053024 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.052771    5293 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c02f9-0059-4a76-a9b2-ed09a9ffb830-maas-api-openshift-ca\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.103122 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.103080    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h" podUID="082b7243-443f-49c1-90ad-955cbe31ec7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:40.103500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.103089    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt" podUID="3cb196ab-0bf0-4a69-9b1f-61328e7a2804" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:40.173626 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.173588    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283c48f8-8987-458f-8d6e-07ea7ab01754" path="/var/lib/kubelet/pods/283c48f8-8987-458f-8d6e-07ea7ab01754/volumes"
Oct 31 02:18:40.174076 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.174027    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46aed14a-2163-408d-bb68-fa6951479193" path="/var/lib/kubelet/pods/46aed14a-2163-408d-bb68-fa6951479193/volumes"
Oct 31 02:18:40.174530 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.174509    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee13e9ec-c271-4076-9849-a843994efad4" path="/var/lib/kubelet/pods/ee13e9ec-c271-4076-9849-a843994efad4/volumes"
Oct 31 02:18:40.176181 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.175381    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22ea351-d8b6-499f-98ed-068aa33808d0" path="/var/lib/kubelet/pods/f22ea351-d8b6-499f-98ed-068aa33808d0/volumes"
Oct 31 02:18:40.305422 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.305386    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 02:18:40.358634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.358588    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 02:18:40.456387 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456354    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-token\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-token\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456401    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-data\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456440    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-kube-api-access-548tq\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-kube-api-access-548tq\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456468    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-envoy\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456520    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-socket\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456564 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456552    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-certs\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456573    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-podinfo\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456592    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa358a0d-fd41-42a0-bbac-52875ea48c46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa358a0d-fd41-42a0-bbac-52875ea48c46-istiod-ca-cert\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456613    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-socket\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.456834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456643    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-envoy\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.456834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456692    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-credential-socket\") pod \"fa358a0d-fd41-42a0-bbac-52875ea48c46\" (UID: \"fa358a0d-fd41-42a0-bbac-52875ea48c46\") "
Oct 31 02:18:40.456834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456715    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-certs\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.456834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456760    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ddaae62-bb16-4c30-926b-57812db835fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ddaae62-bb16-4c30-926b-57812db835fb-istio-podinfo\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.456834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456819    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ddaae62-bb16-4c30-926b-57812db835fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ddaae62-bb16-4c30-926b-57812db835fb-istiod-ca-cert\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.457414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456883    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-data\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.457414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456927    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-istio-token\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-istio-token\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.457414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456942    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-data" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.457414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456954    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-certs" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.457414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456967    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-kube-api-access-lsqzb\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-kube-api-access-lsqzb\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.457414 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.457020    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-credential-socket\") pod \"7ddaae62-bb16-4c30-926b-57812db835fb\" (UID: \"7ddaae62-bb16-4c30-926b-57812db835fb\") "
Oct 31 02:18:40.457757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.457573    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.457757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.457595    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.457757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.456961    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-socket" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.457757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.457639    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa358a0d-fd41-42a0-bbac-52875ea48c46-istiod-ca-cert" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:40.457757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.457223    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-certs" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.457757 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.457430    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-data" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.458263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.457961    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-credential-socket" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.458263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.457952    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-socket" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.458263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.458005    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddaae62-bb16-4c30-926b-57812db835fb-istiod-ca-cert" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:40.458263 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.458218    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-credential-socket" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.459587 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.459551    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-token" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:40.459903 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.459881    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-kube-api-access-548tq" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "kube-api-access-548tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:40.460098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.460071    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-podinfo" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:40.460185 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.460165    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ddaae62-bb16-4c30-926b-57812db835fb-istio-podinfo" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:40.460238 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.460174    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-envoy" pod "fa358a0d-fd41-42a0-bbac-52875ea48c46" (UID: "fa358a0d-fd41-42a0-bbac-52875ea48c46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.460534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.460520    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-envoy" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.460596 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.460537    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-kube-api-access-lsqzb" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "kube-api-access-lsqzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:40.461254 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.461236    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-istio-token" pod "7ddaae62-bb16-4c30-926b-57812db835fb" (UID: "7ddaae62-bb16-4c30-926b-57812db835fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:40.559144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559030    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-548tq\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-kube-api-access-548tq\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559098    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559108    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559117    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559128    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa358a0d-fd41-42a0-bbac-52875ea48c46-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559136    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559148    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559157    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa358a0d-fd41-42a0-bbac-52875ea48c46-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559166    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559174    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ddaae62-bb16-4c30-926b-57812db835fb-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559182    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ddaae62-bb16-4c30-926b-57812db835fb-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559191    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559199    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559207    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsqzb\" (UniqueName: \"kubernetes.io/projected/7ddaae62-bb16-4c30-926b-57812db835fb-kube-api-access-lsqzb\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559217    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ddaae62-bb16-4c30-926b-57812db835fb-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.559479 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.559226    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa358a0d-fd41-42a0-bbac-52875ea48c46-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.661154 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.661123    5293 generic.go:356] "Generic (PLEG): container finished" podID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerID="e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5" exitCode=0
Oct 31 02:18:40.661356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.661187    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"
Oct 31 02:18:40.661356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.661213    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" event={"ID":"fa358a0d-fd41-42a0-bbac-52875ea48c46","Type":"ContainerDied","Data":"e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5"}
Oct 31 02:18:40.661356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.661280    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747" event={"ID":"fa358a0d-fd41-42a0-bbac-52875ea48c46","Type":"ContainerDied","Data":"3a1977b3c8e3af655c66708f82f0dbef8dd443e0805125addf151a9991832c08"}
Oct 31 02:18:40.661356 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.661312    5293 scope.go:122] "RemoveContainer" containerID="e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5"
Oct 31 02:18:40.662724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.662667    5293 generic.go:356] "Generic (PLEG): container finished" podID="7ddaae62-bb16-4c30-926b-57812db835fb" containerID="e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391" exitCode=0
Oct 31 02:18:40.662724 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.662712    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"
Oct 31 02:18:40.662866 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.662755    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" event={"ID":"7ddaae62-bb16-4c30-926b-57812db835fb","Type":"ContainerDied","Data":"e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391"}
Oct 31 02:18:40.662866 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.662799    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms" event={"ID":"7ddaae62-bb16-4c30-926b-57812db835fb","Type":"ContainerDied","Data":"e900aeae9d818b052107b301c7a350e12f1bbc479a5a4b18ec6a9a4f79bab3ee"}
Oct 31 02:18:40.664523 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.664502    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t"
Oct 31 02:18:40.664523 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.664500    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb9a6e2-hsf5t" event={"ID":"6b1c02f9-0059-4a76-a9b2-ed09a9ffb830","Type":"ContainerDied","Data":"e90a876abf054a5ee66ee3c4c70da7465bd9256b7f87b6b5545b3b0e398de603"}
Oct 31 02:18:40.664704 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.664556    5293 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e90a876abf054a5ee66ee3c4c70da7465bd9256b7f87b6b5545b3b0e398de603"
Oct 31 02:18:40.673059 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.672990    5293 scope.go:122] "RemoveContainer" containerID="fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8"
Oct 31 02:18:40.685296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.685278    5293 scope.go:122] "RemoveContainer" containerID="e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5"
Oct 31 02:18:40.685616 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:40.685577    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5\": container with ID starting with e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5 not found: ID does not exist" containerID="e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5"
Oct 31 02:18:40.685616 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.685606    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5"} err="failed to get container status \"e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5\": rpc error: code = NotFound desc = could not find container \"e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5\": container with ID starting with e99d2103dbc1732513d088cbaa8860c75946b7f67ddcc254f8ead2d546b8c9c5 not found: ID does not exist"
Oct 31 02:18:40.685794 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.685623    5293 scope.go:122] "RemoveContainer" containerID="fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8"
Oct 31 02:18:40.685907 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:40.685880    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8\": container with ID starting with fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8 not found: ID does not exist" containerID="fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8"
Oct 31 02:18:40.685952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.685917    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8"} err="failed to get container status \"fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8\": rpc error: code = NotFound desc = could not find container \"fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8\": container with ID starting with fe4c78b7244785fd29479bdfb040e5e5ceb08cbbe6c8f96058042a6f882ebfd8 not found: ID does not exist"
Oct 31 02:18:40.685952 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.685939    5293 scope.go:122] "RemoveContainer" containerID="e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391"
Oct 31 02:18:40.688803 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.688782    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"]
Oct 31 02:18:40.693586 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.693566    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-sc747"]
Oct 31 02:18:40.700231 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.700213    5293 scope.go:122] "RemoveContainer" containerID="e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391"
Oct 31 02:18:40.700514 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:40.700492    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391\": container with ID starting with e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391 not found: ID does not exist" containerID="e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391"
Oct 31 02:18:40.700601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.700520    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391"} err="failed to get container status \"e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391\": rpc error: code = NotFound desc = could not find container \"e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391\": container with ID starting with e3461a1dde7440d81cf557a4f0d114ae468d936df1a390bb41571beeb4145391 not found: ID does not exist"
Oct 31 02:18:40.707272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.707245    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"]
Oct 31 02:18:40.709523 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:40.709504    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-75dms"]
Oct 31 02:18:41.777788 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:41.777756    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:18:41.785940 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:41.785906    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms" podUID="918c7c8a-23e4-4322-862b-d4f7b5de0fc3" containerName="payload-processing" containerID="cri-o://9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25" gracePeriod=25
Oct 31 02:18:42.173286 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:42.173255    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" path="/var/lib/kubelet/pods/7ddaae62-bb16-4c30-926b-57812db835fb/volumes"
Oct 31 02:18:42.173745 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:42.173730    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" path="/var/lib/kubelet/pods/fa358a0d-fd41-42a0-bbac-52875ea48c46/volumes"
Oct 31 02:18:42.202086 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:42.202057    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:18:43.148822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:43.148784    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms" podUID="918c7c8a-23e4-4322-862b-d4f7b5de0fc3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:46.373850 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.373809    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main" containerID="cri-o://cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2" gracePeriod=45
Oct 31 02:18:46.378174 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.378142    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main" containerID="cri-o://09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226" gracePeriod=45
Oct 31 02:18:46.411508 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.411456    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main" containerID="cri-o://298c67d5193793144a5c46685c27b1286f6dbfe524b0f40ba5e5251c3fd411f4" gracePeriod=45
Oct 31 02:18:46.694230 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.694202    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:18:46.697417 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.697398    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:18:46.718487 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.718453    5293 generic.go:356] "Generic (PLEG): container finished" podID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerID="298c67d5193793144a5c46685c27b1286f6dbfe524b0f40ba5e5251c3fd411f4" exitCode=0
Oct 31 02:18:46.718649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.718538    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" event={"ID":"6dcabdce-f8cb-420c-9f80-116695aea37e","Type":"ContainerDied","Data":"298c67d5193793144a5c46685c27b1286f6dbfe524b0f40ba5e5251c3fd411f4"}
Oct 31 02:18:46.722756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.721105    5293 generic.go:356] "Generic (PLEG): container finished" podID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerID="cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2" exitCode=0
Oct 31 02:18:46.722756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.721301    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"
Oct 31 02:18:46.722756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.721419    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" event={"ID":"5990f411-ac1a-405b-ab45-5ea2903a3fa0","Type":"ContainerDied","Data":"cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2"}
Oct 31 02:18:46.722756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.721445    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf" event={"ID":"5990f411-ac1a-405b-ab45-5ea2903a3fa0","Type":"ContainerDied","Data":"de0c34046de2a6474de997be30b8065e8ad90387296a7fa18f56625b19dc80dd"}
Oct 31 02:18:46.722756 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.721466    5293 scope.go:122] "RemoveContainer" containerID="cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2"
Oct 31 02:18:46.726091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.724758    5293 generic.go:356] "Generic (PLEG): container finished" podID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerID="09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226" exitCode=0
Oct 31 02:18:46.726091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.724885    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" event={"ID":"ccb5b30f-e8ae-48ed-beb8-caec78c713f3","Type":"ContainerDied","Data":"09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226"}
Oct 31 02:18:46.726091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.725155    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg" event={"ID":"ccb5b30f-e8ae-48ed-beb8-caec78c713f3","Type":"ContainerDied","Data":"f1cfc0a372f997ea71f181464bbcf9e02905cc2dae9c7c223a64ea3ac4c81ae0"}
Oct 31 02:18:46.726091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.725188    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"
Oct 31 02:18:46.726091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.725609    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:18:46.736834 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.736816    5293 scope.go:122] "RemoveContainer" containerID="cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2"
Oct 31 02:18:46.737192 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:46.737162    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2\": container with ID starting with cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2 not found: ID does not exist" containerID="cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2"
Oct 31 02:18:46.737296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.737201    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2"} err="failed to get container status \"cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2\": rpc error: code = NotFound desc = could not find container \"cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2\": container with ID starting with cefbb687511cbf3f22c4a10d7a1afd3c6173216378bb61072e1291111ede64c2 not found: ID does not exist"
Oct 31 02:18:46.737296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.737225    5293 scope.go:122] "RemoveContainer" containerID="09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226"
Oct 31 02:18:46.754331 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.754311    5293 scope.go:122] "RemoveContainer" containerID="09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226"
Oct 31 02:18:46.755612 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:46.755590    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226\": container with ID starting with 09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226 not found: ID does not exist" containerID="09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226"
Oct 31 02:18:46.755700 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.755625    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226"} err="failed to get container status \"09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226\": rpc error: code = NotFound desc = could not find container \"09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226\": container with ID starting with 09f380978344bcc5c06eaf1b9ce14bb476aa187f2461d19da2637ac93cab5226 not found: ID does not exist"
Oct 31 02:18:46.830009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.829977    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-dshm\") pod \"6dcabdce-f8cb-420c-9f80-116695aea37e\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") "
Oct 31 02:18:46.830009 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830011    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-home\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-home\") pod \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") "
Oct 31 02:18:46.830297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830057    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tmp-dir\") pod \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") "
Oct 31 02:18:46.830297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830114    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-dshm\") pod \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") "
Oct 31 02:18:46.830297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830144    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-model-cache\") pod \"6dcabdce-f8cb-420c-9f80-116695aea37e\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") "
Oct 31 02:18:46.830297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830176    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-tmp-dir\") pod \"6dcabdce-f8cb-420c-9f80-116695aea37e\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") "
Oct 31 02:18:46.830297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830206    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-dshm\") pod \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") "
Oct 31 02:18:46.830297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830267    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tls-certs\") pod \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") "
Oct 31 02:18:46.830651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830304    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-home" pod "5990f411-ac1a-405b-ab45-5ea2903a3fa0" (UID: "5990f411-ac1a-405b-ab45-5ea2903a3fa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.830651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830307    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5990f411-ac1a-405b-ab45-5ea2903a3fa0-kube-api-access-mlvwg\" (UniqueName: \"kubernetes.io/projected/5990f411-ac1a-405b-ab45-5ea2903a3fa0-kube-api-access-mlvwg\") pod \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") "
Oct 31 02:18:46.830651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830424    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dcabdce-f8cb-420c-9f80-116695aea37e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcabdce-f8cb-420c-9f80-116695aea37e-tls-certs\") pod \"6dcabdce-f8cb-420c-9f80-116695aea37e\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") "
Oct 31 02:18:46.830651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830462    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-home\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-home\") pod \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") "
Oct 31 02:18:46.830651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830540    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-model-cache\") pod \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") "
Oct 31 02:18:46.830651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830604    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-home\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-home\") pod \"6dcabdce-f8cb-420c-9f80-116695aea37e\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") "
Oct 31 02:18:46.830651 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830640    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcabdce-f8cb-420c-9f80-116695aea37e-kube-api-access-ghmf7\" (UniqueName: \"kubernetes.io/projected/6dcabdce-f8cb-420c-9f80-116695aea37e-kube-api-access-ghmf7\") pod \"6dcabdce-f8cb-420c-9f80-116695aea37e\" (UID: \"6dcabdce-f8cb-420c-9f80-116695aea37e\") "
Oct 31 02:18:46.831115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830702    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tmp-dir\") pod \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") "
Oct 31 02:18:46.831115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830735    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-model-cache\") pod \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") "
Oct 31 02:18:46.831115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830766    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tls-certs\" (UniqueName: \"kubernetes.io/secret/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tls-certs\") pod \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\" (UID: \"5990f411-ac1a-405b-ab45-5ea2903a3fa0\") "
Oct 31 02:18:46.831115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830797    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-kube-api-access-xrjxz\" (UniqueName: \"kubernetes.io/projected/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-kube-api-access-xrjxz\") pod \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\" (UID: \"ccb5b30f-e8ae-48ed-beb8-caec78c713f3\") "
Oct 31 02:18:46.831115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.830902    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-home" pod "ccb5b30f-e8ae-48ed-beb8-caec78c713f3" (UID: "ccb5b30f-e8ae-48ed-beb8-caec78c713f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.831421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.831212    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-model-cache" pod "6dcabdce-f8cb-420c-9f80-116695aea37e" (UID: "6dcabdce-f8cb-420c-9f80-116695aea37e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.831529 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.831510    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.831599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.831535    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.831599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.831550    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.831861 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.831840    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tmp-dir" pod "ccb5b30f-e8ae-48ed-beb8-caec78c713f3" (UID: "ccb5b30f-e8ae-48ed-beb8-caec78c713f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.832652 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.832551    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tmp-dir" pod "5990f411-ac1a-405b-ab45-5ea2903a3fa0" (UID: "5990f411-ac1a-405b-ab45-5ea2903a3fa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.832924 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.832848    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-model-cache" pod "5990f411-ac1a-405b-ab45-5ea2903a3fa0" (UID: "5990f411-ac1a-405b-ab45-5ea2903a3fa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.833032 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.832950    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-tmp-dir" pod "6dcabdce-f8cb-420c-9f80-116695aea37e" (UID: "6dcabdce-f8cb-420c-9f80-116695aea37e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.833032 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.832996    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-model-cache" pod "ccb5b30f-e8ae-48ed-beb8-caec78c713f3" (UID: "ccb5b30f-e8ae-48ed-beb8-caec78c713f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.833752 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.833416    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-home" pod "6dcabdce-f8cb-420c-9f80-116695aea37e" (UID: "6dcabdce-f8cb-420c-9f80-116695aea37e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.833752 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.833647    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-dshm" pod "6dcabdce-f8cb-420c-9f80-116695aea37e" (UID: "6dcabdce-f8cb-420c-9f80-116695aea37e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.833896 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.833825    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-kube-api-access-xrjxz" pod "ccb5b30f-e8ae-48ed-beb8-caec78c713f3" (UID: "ccb5b30f-e8ae-48ed-beb8-caec78c713f3"). InnerVolumeSpecName "kube-api-access-xrjxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:46.836433 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.836409    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-dshm" pod "ccb5b30f-e8ae-48ed-beb8-caec78c713f3" (UID: "ccb5b30f-e8ae-48ed-beb8-caec78c713f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.836560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.836420    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-dshm" pod "5990f411-ac1a-405b-ab45-5ea2903a3fa0" (UID: "5990f411-ac1a-405b-ab45-5ea2903a3fa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.836560 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.836445    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcabdce-f8cb-420c-9f80-116695aea37e-kube-api-access-ghmf7" pod "6dcabdce-f8cb-420c-9f80-116695aea37e" (UID: "6dcabdce-f8cb-420c-9f80-116695aea37e"). InnerVolumeSpecName "kube-api-access-ghmf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:46.836714 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.836693    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5990f411-ac1a-405b-ab45-5ea2903a3fa0-kube-api-access-mlvwg" pod "5990f411-ac1a-405b-ab45-5ea2903a3fa0" (UID: "5990f411-ac1a-405b-ab45-5ea2903a3fa0"). InnerVolumeSpecName "kube-api-access-mlvwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:46.837293 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.837270    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tls-certs" pod "ccb5b30f-e8ae-48ed-beb8-caec78c713f3" (UID: "ccb5b30f-e8ae-48ed-beb8-caec78c713f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:46.837736 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.837715    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcabdce-f8cb-420c-9f80-116695aea37e-tls-certs" pod "6dcabdce-f8cb-420c-9f80-116695aea37e" (UID: "6dcabdce-f8cb-420c-9f80-116695aea37e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:46.838020 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.838002    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tls-certs" pod "5990f411-ac1a-405b-ab45-5ea2903a3fa0" (UID: "5990f411-ac1a-405b-ab45-5ea2903a3fa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:46.933091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933024    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcabdce-f8cb-420c-9f80-116695aea37e-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933091 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933093    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933107    5293 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-home\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933151    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmf7\" (UniqueName: \"kubernetes.io/projected/6dcabdce-f8cb-420c-9f80-116695aea37e-kube-api-access-ghmf7\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933167    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933179    5293 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-model-cache\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933190    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5990f411-ac1a-405b-ab45-5ea2903a3fa0-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933203    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjxz\" (UniqueName: \"kubernetes.io/projected/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-kube-api-access-xrjxz\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933219    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933231    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933242    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990f411-ac1a-405b-ab45-5ea2903a3fa0-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933253    5293 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcabdce-f8cb-420c-9f80-116695aea37e-tmp-dir\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933264    5293 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-dshm\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933275    5293 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb5b30f-e8ae-48ed-beb8-caec78c713f3-tls-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.933370 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:46.933287    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvwg\" (UniqueName: \"kubernetes.io/projected/5990f411-ac1a-405b-ab45-5ea2903a3fa0-kube-api-access-mlvwg\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:47.045653 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.045623    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"]
Oct 31 02:18:47.049621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.049599    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-2-w3-7e5955-kserve-6dc898bb45-cn5rf"]
Oct 31 02:18:47.059992 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.059969    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"]
Oct 31 02:18:47.065556 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.065533    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/distinct-w3-7e5955-kserve-6d49bb8bd6-6crxg"]
Oct 31 02:18:47.192103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.192015    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"]
Oct 31 02:18:47.192269 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.192244    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" containerID="cri-o://06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054" gracePeriod=30
Oct 31 02:18:47.734535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.734507    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"
Oct 31 02:18:47.734535 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.734515    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n" event={"ID":"6dcabdce-f8cb-420c-9f80-116695aea37e","Type":"ContainerDied","Data":"082c9801e043ef6fb9fc4997709182cb18bc5536e957e536f3d593c6323f2030"}
Oct 31 02:18:47.735133 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.734573    5293 scope.go:122] "RemoveContainer" containerID="298c67d5193793144a5c46685c27b1286f6dbfe524b0f40ba5e5251c3fd411f4"
Oct 31 02:18:47.757668 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.757641    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"]
Oct 31 02:18:47.762475 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:47.762451    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/simulator-w3-7e5955-kserve-67fcf8454b-72v9n"]
Oct 31 02:18:48.173476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:48.173431    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" path="/var/lib/kubelet/pods/5990f411-ac1a-405b-ab45-5ea2903a3fa0/volumes"
Oct 31 02:18:48.173959 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:48.173937    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" path="/var/lib/kubelet/pods/6dcabdce-f8cb-420c-9f80-116695aea37e/volumes"
Oct 31 02:18:48.174421 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:48.174402    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" path="/var/lib/kubelet/pods/ccb5b30f-e8ae-48ed-beb8-caec78c713f3/volumes"
Oct 31 02:18:49.369030 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.368961    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 02:18:49.425095 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.425071    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 02:18:49.562552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.562522    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-plugins-config-volume\") pod \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\" (UID: \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\") "
Oct 31 02:18:49.562728 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.562638    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/082b7243-443f-49c1-90ad-955cbe31ec7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082b7243-443f-49c1-90ad-955cbe31ec7f-plugins-config-volume\") pod \"082b7243-443f-49c1-90ad-955cbe31ec7f\" (UID: \"082b7243-443f-49c1-90ad-955cbe31ec7f\") "
Oct 31 02:18:49.562728 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.562693    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-kube-api-access-hpgtr\" (UniqueName: \"kubernetes.io/projected/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-kube-api-access-hpgtr\") pod \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\" (UID: \"3cb196ab-0bf0-4a69-9b1f-61328e7a2804\") "
Oct 31 02:18:49.562883 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.562760    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082b7243-443f-49c1-90ad-955cbe31ec7f-kube-api-access-wkfpl\" (UniqueName: \"kubernetes.io/projected/082b7243-443f-49c1-90ad-955cbe31ec7f-kube-api-access-wkfpl\") pod \"082b7243-443f-49c1-90ad-955cbe31ec7f\" (UID: \"082b7243-443f-49c1-90ad-955cbe31ec7f\") "
Oct 31 02:18:49.562971 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.562949    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-plugins-config-volume" pod "3cb196ab-0bf0-4a69-9b1f-61328e7a2804" (UID: "3cb196ab-0bf0-4a69-9b1f-61328e7a2804"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:49.563102 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.563075    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082b7243-443f-49c1-90ad-955cbe31ec7f-plugins-config-volume" pod "082b7243-443f-49c1-90ad-955cbe31ec7f" (UID: "082b7243-443f-49c1-90ad-955cbe31ec7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:49.563234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.563219    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:49.563297 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.563237    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082b7243-443f-49c1-90ad-955cbe31ec7f-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:49.565076 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.565033    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-kube-api-access-hpgtr" pod "3cb196ab-0bf0-4a69-9b1f-61328e7a2804" (UID: "3cb196ab-0bf0-4a69-9b1f-61328e7a2804"). InnerVolumeSpecName "kube-api-access-hpgtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:49.565135 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.565112    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082b7243-443f-49c1-90ad-955cbe31ec7f-kube-api-access-wkfpl" pod "082b7243-443f-49c1-90ad-955cbe31ec7f" (UID: "082b7243-443f-49c1-90ad-955cbe31ec7f"). InnerVolumeSpecName "kube-api-access-wkfpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:49.664608 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.664574    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpgtr\" (UniqueName: \"kubernetes.io/projected/3cb196ab-0bf0-4a69-9b1f-61328e7a2804-kube-api-access-hpgtr\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:49.664608 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.664604    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfpl\" (UniqueName: \"kubernetes.io/projected/082b7243-443f-49c1-90ad-955cbe31ec7f-kube-api-access-wkfpl\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:49.757599 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.757563    5293 generic.go:356] "Generic (PLEG): container finished" podID="3cb196ab-0bf0-4a69-9b1f-61328e7a2804" containerID="5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8" exitCode=0
Oct 31 02:18:49.757795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.757627    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"
Oct 31 02:18:49.757795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.757661    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt" event={"ID":"3cb196ab-0bf0-4a69-9b1f-61328e7a2804","Type":"ContainerDied","Data":"5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8"}
Oct 31 02:18:49.757795 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.757747    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt" event={"ID":"3cb196ab-0bf0-4a69-9b1f-61328e7a2804","Type":"ContainerDied","Data":"d8e7cff5f326f91c9cc37b1e615988f9f409cf9b156a78e72bb93df9c084948f"}
Oct 31 02:18:49.757985 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.757811    5293 scope.go:122] "RemoveContainer" containerID="5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8"
Oct 31 02:18:49.759698 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.759667    5293 generic.go:356] "Generic (PLEG): container finished" podID="082b7243-443f-49c1-90ad-955cbe31ec7f" containerID="b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2" exitCode=0
Oct 31 02:18:49.759871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.759708    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h" event={"ID":"082b7243-443f-49c1-90ad-955cbe31ec7f","Type":"ContainerDied","Data":"b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2"}
Oct 31 02:18:49.759871 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.759751    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h" event={"ID":"082b7243-443f-49c1-90ad-955cbe31ec7f","Type":"ContainerDied","Data":"645a41c3b06de941bb1818b66739fa093277937d5a25ac97255578f93088b17c"}
Oct 31 02:18:49.760137 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.760113    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"
Oct 31 02:18:49.774574 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.774546    5293 scope.go:122] "RemoveContainer" containerID="5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8"
Oct 31 02:18:49.777374 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:49.777342    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8\": container with ID starting with 5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8 not found: ID does not exist" containerID="5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8"
Oct 31 02:18:49.777471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.777382    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8"} err="failed to get container status \"5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8\": rpc error: code = NotFound desc = could not find container \"5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8\": container with ID starting with 5bd3f1d5d69aebc1bab11bc17456a2a5d4ab4415f60c3d0fe7c3942e7a2a38e8 not found: ID does not exist"
Oct 31 02:18:49.777471 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.777399    5293 scope.go:122] "RemoveContainer" containerID="b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2"
Oct 31 02:18:49.787223 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.787205    5293 scope.go:122] "RemoveContainer" containerID="b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2"
Oct 31 02:18:49.787508 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:49.787489    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2\": container with ID starting with b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2 not found: ID does not exist" containerID="b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2"
Oct 31 02:18:49.787561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.787516    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2"} err="failed to get container status \"b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2\": rpc error: code = NotFound desc = could not find container \"b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2\": container with ID starting with b9f7ee0d669f5a4847b4ca4a73f4ed16bc8156628c46ef7819c27ab895e3b9e2 not found: ID does not exist"
Oct 31 02:18:49.792420 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.792400    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"]
Oct 31 02:18:49.797429 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.797407    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6ae519-5979494f57-gwzrt"]
Oct 31 02:18:49.807901 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.807877    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"]
Oct 31 02:18:49.814310 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:49.814289    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6ae519-c9bf8d8cf-jlf9h"]
Oct 31 02:18:50.173518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:50.173484    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082b7243-443f-49c1-90ad-955cbe31ec7f" path="/var/lib/kubelet/pods/082b7243-443f-49c1-90ad-955cbe31ec7f/volumes"
Oct 31 02:18:50.173889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:50.173875    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb196ab-0bf0-4a69-9b1f-61328e7a2804" path="/var/lib/kubelet/pods/3cb196ab-0bf0-4a69-9b1f-61328e7a2804/volumes"
Oct 31 02:18:50.947534 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:50.947499    5293 scope.go:122] "RemoveContainer" containerID="039e900f8f3fbdf50f9b6ab4a608f3048cfba22d7027c07cd6380f2db85ce998"
Oct 31 02:18:50.958864 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:50.958831    5293 scope.go:122] "RemoveContainer" containerID="614f0b02914173bc4cce4cb4fc37a5fb143c09acd1a06fc77c285b58a94fd92c"
Oct 31 02:18:50.969500 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:50.969475    5293 scope.go:122] "RemoveContainer" containerID="ddf7aaddeb829506dd7c73efd1b88486bf544ea391dd9ba91f4c464594d7dbba"
Oct 31 02:18:51.027418 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:51.027332    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"]
Oct 31 02:18:51.029708 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:51.029685    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c463dc5-7gxmm"]
Oct 31 02:18:51.226386 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:51.226302    5293 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:18:51.226551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:51.226416    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:18:51.698631 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:51.698596    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-786vs"]
Oct 31 02:18:52.172256 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.172226    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7920f04-647f-4a86-a616-c0bebac0a084" path="/var/lib/kubelet/pods/f7920f04-647f-4a86-a616-c0bebac0a084/volumes"
Oct 31 02:18:52.464838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.464816    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 02:18:52.504193 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504154    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-socket\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504193 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504192    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-envoy\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504214    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-token\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-token\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504231    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-data\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504434 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504391    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-credential-socket\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504433    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-certs\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504466    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-kube-api-access-82fn8\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-kube-api-access-82fn8\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504545    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-podinfo\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504601 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504530    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-socket" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.504837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504603    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f225a7a3-dddd-41c3-b5fd-b6806209befd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225a7a3-dddd-41c3-b5fd-b6806209befd-istiod-ca-cert\") pod \"f225a7a3-dddd-41c3-b5fd-b6806209befd\" (UID: \"f225a7a3-dddd-41c3-b5fd-b6806209befd\") "
Oct 31 02:18:52.504837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504695    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-credential-socket" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.504837 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.504769    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-certs" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.505125 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.505103    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.505195 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.505131    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.505195 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.505146    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.505473 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.505451    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f225a7a3-dddd-41c3-b5fd-b6806209befd-istiod-ca-cert" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:52.505539 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.505522    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-data" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.507002 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.506978    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-token" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:52.507552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.507527    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-envoy" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.507552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.507543    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-kube-api-access-82fn8" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "kube-api-access-82fn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:52.507722 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.507697    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-podinfo" pod "f225a7a3-dddd-41c3-b5fd-b6806209befd" (UID: "f225a7a3-dddd-41c3-b5fd-b6806209befd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:52.605884 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.605851    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82fn8\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-kube-api-access-82fn8\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.605884 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.605879    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.605884 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.605889    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225a7a3-dddd-41c3-b5fd-b6806209befd-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.606164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.605898    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.606164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.605907    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.606164 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.605916    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225a7a3-dddd-41c3-b5fd-b6806209befd-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.816911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.816808    5293 generic.go:356] "Generic (PLEG): container finished" podID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerID="06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054" exitCode=0
Oct 31 02:18:52.816911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.816858    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"
Oct 31 02:18:52.816911 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.816897    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" event={"ID":"f225a7a3-dddd-41c3-b5fd-b6806209befd","Type":"ContainerDied","Data":"06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054"}
Oct 31 02:18:52.817245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.816956    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5" event={"ID":"f225a7a3-dddd-41c3-b5fd-b6806209befd","Type":"ContainerDied","Data":"3555f0068a0f415cc06e54ce3baf00d25a10aeda487f512271cb360d91b614af"}
Oct 31 02:18:52.817245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.816979    5293 scope.go:122] "RemoveContainer" containerID="06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054"
Oct 31 02:18:52.828231 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.828210    5293 scope.go:122] "RemoveContainer" containerID="06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054"
Oct 31 02:18:52.828535 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:18:52.828512    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054\": container with ID starting with 06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054 not found: ID does not exist" containerID="06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054"
Oct 31 02:18:52.828608 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.828549    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054"} err="failed to get container status \"06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054\": rpc error: code = NotFound desc = could not find container \"06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054\": container with ID starting with 06f8997d24b9bb2441173ed7991d41065194854a831475e26b0d245b34382054 not found: ID does not exist"
Oct 31 02:18:52.841145 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.841119    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"]
Oct 31 02:18:52.847615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:52.847592    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-k49d5"]
Oct 31 02:18:53.149203 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:53.149169    5293 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms" podUID="918c7c8a-23e4-4322-862b-d4f7b5de0fc3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:54.172518 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:54.172488    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" path="/var/lib/kubelet/pods/f225a7a3-dddd-41c3-b5fd-b6806209befd/volumes"
Oct 31 02:18:58.390989 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:58.390957    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"]
Oct 31 02:18:58.391462 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:18:58.391229    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy" containerID="cri-o://a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a" gracePeriod=30
Oct 31 02:19:00.259219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:00.259189    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8wgfx_828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba/kuadrant-console-plugin/0.log"
Oct 31 02:19:00.946758 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:00.946713    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-46qd9_771a7056-0944-443b-969c-6d4283bec9bf/manager/0.log"
Oct 31 02:19:01.208637 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:01.208560    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-786vs_d61f77a9-2b32-463f-bc28-64e21cd43a7b/limitador/0.log"
Oct 31 02:19:02.032219 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.032188    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 02:19:02.099349 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.099277    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-plugins-config-volume\") pod \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\" (UID: \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\") "
Oct 31 02:19:02.099483 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.099357    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-kube-api-access-22ffv\" (UniqueName: \"kubernetes.io/projected/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-kube-api-access-22ffv\") pod \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\" (UID: \"918c7c8a-23e4-4322-862b-d4f7b5de0fc3\") "
Oct 31 02:19:02.099619 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.099598    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-plugins-config-volume" pod "918c7c8a-23e4-4322-862b-d4f7b5de0fc3" (UID: "918c7c8a-23e4-4322-862b-d4f7b5de0fc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:19:02.101554 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.101525    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-kube-api-access-22ffv" pod "918c7c8a-23e4-4322-862b-d4f7b5de0fc3" (UID: "918c7c8a-23e4-4322-862b-d4f7b5de0fc3"). InnerVolumeSpecName "kube-api-access-22ffv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:02.201375 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.201344    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22ffv\" (UniqueName: \"kubernetes.io/projected/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-kube-api-access-22ffv\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:02.201513 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.201437    5293 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c7c8a-23e4-4322-862b-d4f7b5de0fc3-plugins-config-volume\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:02.892378 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.892342    5293 generic.go:356] "Generic (PLEG): container finished" podID="918c7c8a-23e4-4322-862b-d4f7b5de0fc3" containerID="9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25" exitCode=0
Oct 31 02:19:02.892559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.892385    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"
Oct 31 02:19:02.892559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.892428    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms" event={"ID":"918c7c8a-23e4-4322-862b-d4f7b5de0fc3","Type":"ContainerDied","Data":"9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25"}
Oct 31 02:19:02.892559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.892479    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms" event={"ID":"918c7c8a-23e4-4322-862b-d4f7b5de0fc3","Type":"ContainerDied","Data":"edc3c1788de7adcc256028761f19273c1cf38f27e50f9cfc737c1b05e9bc9d85"}
Oct 31 02:19:02.892559 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.892506    5293 scope.go:122] "RemoveContainer" containerID="9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25"
Oct 31 02:19:02.902226 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.902208    5293 scope.go:122] "RemoveContainer" containerID="9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25"
Oct 31 02:19:02.902493 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:19:02.902475    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25\": container with ID starting with 9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25 not found: ID does not exist" containerID="9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25"
Oct 31 02:19:02.902545 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.902501    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25"} err="failed to get container status \"9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25\": rpc error: code = NotFound desc = could not find container \"9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25\": container with ID starting with 9e4a7a87303b465565eb80dd46489733316d16ee1ff469d97fd2bb6998a1bc25 not found: ID does not exist"
Oct 31 02:19:02.909198 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.909177    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"]
Oct 31 02:19:02.913900 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:02.913881    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7e5955-54cd86c7cc-r57ms"]
Oct 31 02:19:03.021717 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.021686    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"]
Oct 31 02:19:03.024633 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.024605    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4681fa31-vdk8b"]
Oct 31 02:19:03.489412 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.489385    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s9rxb_e75104bf-ae55-488f-bab5-595defac1e17/server/0.log"
Oct 31 02:19:03.646991 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.646965    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 02:19:03.690563 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.690500    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4bjhg_ba3ae4d1-4d61-457c-a552-b4001c770117/manager/0.log"
Oct 31 02:19:03.717710 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.717684    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-kube-api-access-c5d2s\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-kube-api-access-c5d2s\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.717861 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.717728    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-certs\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.717861 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.717751    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istiod-ca-cert\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.718056 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.717916    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-podinfo\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.718191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718068    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-certs" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.718191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718153    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-credential-socket\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.718191 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718183    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istiod-ca-cert" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:19:03.718381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718196    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-socket\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.718381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718282    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-token\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-token\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.718381 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718359    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-data\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.718552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718388    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-envoy\") pod \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\" (UID: \"d46cf4fa-53a4-405b-b669-85f5fce9e3e9\") "
Oct 31 02:19:03.718552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718447    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-credential-socket" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.718552 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718521    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-socket" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.718784 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718633    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-data" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.718993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718890    5293 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.718993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718922    5293 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istiod-ca-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.718993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718936    5293 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-credential-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.718993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718951    5293 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-workload-socket\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.718993 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.718966    5293 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-data\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.721822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.721778    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-token" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:03.721822 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.721809    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-podinfo" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:19:03.722774 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.722745    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-kube-api-access-c5d2s" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "kube-api-access-c5d2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:03.722892 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.722797    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-envoy" pod "d46cf4fa-53a4-405b-b669-85f5fce9e3e9" (UID: "d46cf4fa-53a4-405b-b669-85f5fce9e3e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.820287 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.820241    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.820287 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.820280    5293 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-envoy\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.820287 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.820291    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5d2s\" (UniqueName: \"kubernetes.io/projected/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-kube-api-access-c5d2s\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.820287 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.820301    5293 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46cf4fa-53a4-405b-b669-85f5fce9e3e9-istio-podinfo\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.901938 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.901906    5293 generic.go:356] "Generic (PLEG): container finished" podID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerID="a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a" exitCode=0
Oct 31 02:19:03.902113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.901954    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"
Oct 31 02:19:03.902113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.901986    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" event={"ID":"d46cf4fa-53a4-405b-b669-85f5fce9e3e9","Type":"ContainerDied","Data":"a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a"}
Oct 31 02:19:03.902113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.902056    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z" event={"ID":"d46cf4fa-53a4-405b-b669-85f5fce9e3e9","Type":"ContainerDied","Data":"3e1b18a3480b874e653d4d8339610485d42e5f83bebd4f7f7a3fbca81848f417"}
Oct 31 02:19:03.902113 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.902078    5293 scope.go:122] "RemoveContainer" containerID="a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a"
Oct 31 02:19:03.912319 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.912303    5293 scope.go:122] "RemoveContainer" containerID="a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a"
Oct 31 02:19:03.912563 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:19:03.912541    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a\": container with ID starting with a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a not found: ID does not exist" containerID="a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a"
Oct 31 02:19:03.912615 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.912571    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a"} err="failed to get container status \"a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a\": rpc error: code = NotFound desc = could not find container \"a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a\": container with ID starting with a0d6c40495ca18ba1fa9bd4a845583564ea63f77bc5aebbb1ba63aa38fbfb11a not found: ID does not exist"
Oct 31 02:19:03.924017 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.923996    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"]
Oct 31 02:19:03.928103 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:03.928080    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-58b8z"]
Oct 31 02:19:04.172083 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:04.172032    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8428ea83-43a2-4f98-a98c-2ac754528306" path="/var/lib/kubelet/pods/8428ea83-43a2-4f98-a98c-2ac754528306/volumes"
Oct 31 02:19:04.172430 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:04.172417    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918c7c8a-23e4-4322-862b-d4f7b5de0fc3" path="/var/lib/kubelet/pods/918c7c8a-23e4-4322-862b-d4f7b5de0fc3/volumes"
Oct 31 02:19:04.172792 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:04.172781    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" path="/var/lib/kubelet/pods/d46cf4fa-53a4-405b-b669-85f5fce9e3e9/volumes"
Oct 31 02:19:07.022995 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:07.022948    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"]
Oct 31 02:19:07.027357 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:07.027330    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"]
Oct 31 02:19:07.030114 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:07.030086    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a020f30-rsthb"]
Oct 31 02:19:07.030910 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:07.030893    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2fbf65-95bkj"]
Oct 31 02:19:08.175747 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:08.175704    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f13b80-a936-4d63-aa4b-7a97204b84bc" path="/var/lib/kubelet/pods/23f13b80-a936-4d63-aa4b-7a97204b84bc/volumes"
Oct 31 02:19:08.176468 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:08.176371    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bca3a4-8f28-461a-a742-d6a2c47afcef" path="/var/lib/kubelet/pods/70bca3a4-8f28-461a-a742-d6a2c47afcef/volumes"
Oct 31 02:19:11.777395 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:19:11.777365    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:19:17.040001 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:17.039972    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s9rxb_e75104bf-ae55-488f-bab5-595defac1e17/server/0.log"
Oct 31 02:19:17.231366 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:17.231336    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4bjhg_ba3ae4d1-4d61-457c-a552-b4001c770117/manager/0.log"
Oct 31 02:19:18.425633 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:18.425605    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0f4ecdd9-8xw62_7ef03134-9a32-466a-a379-e533ba5e8a5f/revoke-keys/0.log"
Oct 31 02:19:18.613695 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:18.613666    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-17e82121-kwc96_e6538fad-9fb0-42e3-9e97-d54de4ebafc6/revoke-keys/0.log"
Oct 31 02:19:18.970804 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:18.970767    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5328a1f0-cftbg_512bffc0-b64c-461c-a374-4530f745567f/revoke-keys/0.log"
Oct 31 02:19:19.149804 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:19.149766    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-57403e6a-5pfsz_26eb0659-7c39-451d-8e0b-7c58b85908a8/revoke-keys/0.log"
Oct 31 02:19:19.516115 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:19.516081    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-69a9ade1-89jmz_8ed586a7-b174-4a71-9802-24e4880802d8/revoke-keys/0.log"
Oct 31 02:19:19.698274 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:19.698235    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8fb9a6e2-hsf5t_6b1c02f9-0059-4a76-a9b2-ed09a9ffb830/revoke-keys/0.log"
Oct 31 02:19:20.074350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:20.074318    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a19e4613-qkmrh_415bd3ec-d41e-4ee9-94c2-340f52c1b406/revoke-keys/0.log"
Oct 31 02:19:20.264148 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:20.264118    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aa465b6e-znnsx_044d45ed-06e4-41a3-bb44-bea63b2db1e7/revoke-keys/0.log"
Oct 31 02:19:20.458477 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:20.458450    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fae3dbe7-w9lns_042bc481-4c8e-4a47-b767-09b3a10716e4/revoke-keys/0.log"
Oct 31 02:19:20.642581 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:20.642548    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fca7cb37-nvzcr_e72d05ac-fcfe-450a-9656-772d37af4b65/revoke-keys/0.log"
Oct 31 02:19:22.933492 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:22.933419    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8wgfx_828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba/kuadrant-console-plugin/0.log"
Oct 31 02:19:23.634677 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:23.634639    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-46qd9_771a7056-0944-443b-969c-6d4283bec9bf/manager/0.log"
Oct 31 02:19:23.886072 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:23.885981    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-786vs_d61f77a9-2b32-463f-bc28-64e21cd43a7b/limitador/0.log"
Oct 31 02:19:25.073282 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:25.073252    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5p96n_1b6a6c52-4b49-4647-b86f-b2a91537032c/discovery/0.log"
Oct 31 02:19:25.310430 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:25.310401    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-chwsm_46307ab4-5e52-4d6d-a527-9e7756732832/discovery/0.log"
Oct 31 02:19:26.046634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:26.046558    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b44bd95d7-tdrn5_d6956b96-691a-445b-89e2-2fe5b289674e/kube-auth-proxy/0.log"
Oct 31 02:19:26.427786 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:26.427749    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-6dz48_774c137c-c103-4d59-b525-37ad6df863e5/istio-proxy/0.log"
Oct 31 02:19:27.044104 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:27.044071    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-d8xb2_0cef45b4-73c6-463c-a1e0-716629cc73b0/istio-proxy/0.log"
Oct 31 02:19:27.615688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:27.615653    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-v46nb_d67d9b4f-a0df-43a7-97b7-88df20e0aa19/istio-proxy/0.log"
Oct 31 02:19:27.818672 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:27.818644    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-x47hr_a36f3ee8-e967-4465-a485-7a9b040e02c2/istio-proxy/0.log"
Oct 31 02:19:28.932572 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:28.932539    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-cxqrw_da6a4c1a-a546-4493-8003-6e4fbc1e84a3/payload-pre-processing/0.log"
Oct 31 02:19:30.335241 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:30.335210    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-t4mgd_a1e82e90-a1d5-453d-ac98-c37d03673798/main/0.log"
Oct 31 02:19:30.885499 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:30.885469    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-cfzj2_3760c544-14ca-4dfd-9527-5054edd99b12/main/0.log"
Oct 31 02:19:31.068932 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:31.068904    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pg4v_173fa9c9-e8df-483f-be0b-dd7cb6ba6a7b/main/0.log"
Oct 31 02:19:31.615621 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:31.615594    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-q4rgw_808265a9-b1bb-4c13-ba57-6d4aa0e91931/main/0.log"
Oct 31 02:19:41.777198 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:19:41.777156    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:19:42.283858 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.283824    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfpsd/must-gather-9dvk9"]
Oct 31 02:19:42.284214 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284201    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f22ea351-d8b6-499f-98ed-068aa33808d0" containerName="payload-processing"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284215    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22ea351-d8b6-499f-98ed-068aa33808d0" containerName="payload-processing"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284233    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ee13e9ec-c271-4076-9849-a843994efad4" containerName="payload-pre-processing"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284238    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee13e9ec-c271-4076-9849-a843994efad4" containerName="payload-pre-processing"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284247    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284252    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284259    5293 cpu_manager.go:414] "containerMap: removing container" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284263    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284270    5293 cpu_manager.go:414] "containerMap: removing container" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy"
Oct 31 02:19:42.284272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284275    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284281    5293 cpu_manager.go:414] "containerMap: removing container" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284285    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284290    5293 cpu_manager.go:414] "containerMap: removing container" podUID="918c7c8a-23e4-4322-862b-d4f7b5de0fc3" containerName="payload-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284294    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918c7c8a-23e4-4322-862b-d4f7b5de0fc3" containerName="payload-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284300    5293 cpu_manager.go:414] "containerMap: removing container" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284305    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284313    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6b1c02f9-0059-4a76-a9b2-ed09a9ffb830" containerName="revoke-keys"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284317    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1c02f9-0059-4a76-a9b2-ed09a9ffb830" containerName="revoke-keys"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284323    5293 cpu_manager.go:414] "containerMap: removing container" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284327    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284340    5293 cpu_manager.go:414] "containerMap: removing container" podUID="082b7243-443f-49c1-90ad-955cbe31ec7f" containerName="payload-pre-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284344    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082b7243-443f-49c1-90ad-955cbe31ec7f" containerName="payload-pre-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284352    5293 cpu_manager.go:414] "containerMap: removing container" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284357    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284364    5293 cpu_manager.go:414] "containerMap: removing container" podUID="3cb196ab-0bf0-4a69-9b1f-61328e7a2804" containerName="payload-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284368    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb196ab-0bf0-4a69-9b1f-61328e7a2804" containerName="payload-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284376    5293 cpu_manager.go:414] "containerMap: removing container" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284380    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284433    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb196ab-0bf0-4a69-9b1f-61328e7a2804" containerName="payload-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284443    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="5990f411-ac1a-405b-ab45-5ea2903a3fa0" containerName="main"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284450    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22ea351-d8b6-499f-98ed-068aa33808d0" containerName="payload-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284459    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee13e9ec-c271-4076-9849-a843994efad4" containerName="payload-pre-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284464    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284469    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="f225a7a3-dddd-41c3-b5fd-b6806209befd" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284474    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa358a0d-fd41-42a0-bbac-52875ea48c46" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284480    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddaae62-bb16-4c30-926b-57812db835fb" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284484    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="918c7c8a-23e4-4322-862b-d4f7b5de0fc3" containerName="payload-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284490    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46cf4fa-53a4-405b-b669-85f5fce9e3e9" containerName="istio-proxy"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284495    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1c02f9-0059-4a76-a9b2-ed09a9ffb830" containerName="revoke-keys"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284501    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcabdce-f8cb-420c-9f80-116695aea37e" containerName="main"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284506    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="082b7243-443f-49c1-90ad-955cbe31ec7f" containerName="payload-pre-processing"
Oct 31 02:19:42.284561 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.284511    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb5b30f-e8ae-48ed-beb8-caec78c713f3" containerName="main"
Oct 31 02:19:42.287833 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.287815    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfpsd/must-gather-9dvk9"
Oct 31 02:19:42.297231 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.297205    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfpsd/must-gather-9dvk9"]
Oct 31 02:19:42.302551 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.302528    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfpsd\"/\"openshift-service-ca.crt\""
Oct 31 02:19:42.302869 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.302850    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfpsd\"/\"kube-root-ca.crt\""
Oct 31 02:19:42.303444 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.303426    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfpsd\"/\"default-dockercfg-4k2q4\""
Oct 31 02:19:42.398627 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.398585    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqls\" (UniqueName: \"kubernetes.io/projected/250a107d-9b15-4ac7-8168-b0d40e29b013-kube-api-access-9tqls\") pod \"must-gather-9dvk9\" (UID: \"250a107d-9b15-4ac7-8168-b0d40e29b013\") " pod="openshift-must-gather-zfpsd/must-gather-9dvk9"
Oct 31 02:19:42.398839 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.398636    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/250a107d-9b15-4ac7-8168-b0d40e29b013-must-gather-output\") pod \"must-gather-9dvk9\" (UID: \"250a107d-9b15-4ac7-8168-b0d40e29b013\") " pod="openshift-must-gather-zfpsd/must-gather-9dvk9"
Oct 31 02:19:42.499810 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.499768    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqls\" (UniqueName: \"kubernetes.io/projected/250a107d-9b15-4ac7-8168-b0d40e29b013-kube-api-access-9tqls\") pod \"must-gather-9dvk9\" (UID: \"250a107d-9b15-4ac7-8168-b0d40e29b013\") " pod="openshift-must-gather-zfpsd/must-gather-9dvk9"
Oct 31 02:19:42.499810 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.499816    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/250a107d-9b15-4ac7-8168-b0d40e29b013-must-gather-output\") pod \"must-gather-9dvk9\" (UID: \"250a107d-9b15-4ac7-8168-b0d40e29b013\") " pod="openshift-must-gather-zfpsd/must-gather-9dvk9"
Oct 31 02:19:42.500231 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.500208    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/250a107d-9b15-4ac7-8168-b0d40e29b013-must-gather-output\") pod \"must-gather-9dvk9\" (UID: \"250a107d-9b15-4ac7-8168-b0d40e29b013\") " pod="openshift-must-gather-zfpsd/must-gather-9dvk9"
Oct 31 02:19:42.517350 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.517321    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfpsd\"/\"kube-root-ca.crt\""
Oct 31 02:19:42.528261 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.528242    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfpsd\"/\"openshift-service-ca.crt\""
Oct 31 02:19:42.539084 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.539023    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqls\" (UniqueName: \"kubernetes.io/projected/250a107d-9b15-4ac7-8168-b0d40e29b013-kube-api-access-9tqls\") pod \"must-gather-9dvk9\" (UID: \"250a107d-9b15-4ac7-8168-b0d40e29b013\") " pod="openshift-must-gather-zfpsd/must-gather-9dvk9"
Oct 31 02:19:42.606377 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.606349    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfpsd\"/\"default-dockercfg-4k2q4\""
Oct 31 02:19:42.614459 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.614440    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfpsd/must-gather-9dvk9"
Oct 31 02:19:42.891565 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.891538    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfpsd/must-gather-9dvk9"]
Oct 31 02:19:42.893137 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:19:42.893108    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod250a107d_9b15_4ac7_8168_b0d40e29b013.slice/crio-620b5d319492dce979a30d9a023e3593eb1a0b466236f03a0f3477d573a7b9ab WatchSource:0}: Error finding container 620b5d319492dce979a30d9a023e3593eb1a0b466236f03a0f3477d573a7b9ab: Status 404 returned error can't find the container with id 620b5d319492dce979a30d9a023e3593eb1a0b466236f03a0f3477d573a7b9ab
Oct 31 02:19:42.894893 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:42.894878    5293 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:19:43.151383 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:43.151352    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfpsd/must-gather-9dvk9" event={"ID":"250a107d-9b15-4ac7-8168-b0d40e29b013","Type":"ContainerStarted","Data":"620b5d319492dce979a30d9a023e3593eb1a0b466236f03a0f3477d573a7b9ab"}
Oct 31 02:19:44.429708 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.429675    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"]
Oct 31 02:19:44.430287 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.429904    5293 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm" podUID="46307ab4-5e52-4d6d-a527-9e7756732832" containerName="discovery" containerID="cri-o://9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d" gracePeriod=30
Oct 31 02:19:44.688476 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.688413    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 02:19:44.822178 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.822145    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-cacerts\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-cacerts\") pod \"46307ab4-5e52-4d6d-a527-9e7756732832\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") "
Oct 31 02:19:44.822425 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.822248    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-kube-api-access-dlw2h\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-kube-api-access-dlw2h\") pod \"46307ab4-5e52-4d6d-a527-9e7756732832\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") "
Oct 31 02:19:44.822425 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.822318    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-ca-configmap\") pod \"46307ab4-5e52-4d6d-a527-9e7756732832\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") "
Oct 31 02:19:44.822425 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.822356    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46307ab4-5e52-4d6d-a527-9e7756732832-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46307ab4-5e52-4d6d-a527-9e7756732832-local-certs\") pod \"46307ab4-5e52-4d6d-a527-9e7756732832\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") "
Oct 31 02:19:44.822623 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.822555    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-dns-cert\") pod \"46307ab4-5e52-4d6d-a527-9e7756732832\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") "
Oct 31 02:19:44.822688 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.822643    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-kubeconfig\") pod \"46307ab4-5e52-4d6d-a527-9e7756732832\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") "
Oct 31 02:19:44.822744 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.822682    5293 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-istio-token\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-istio-token\") pod \"46307ab4-5e52-4d6d-a527-9e7756732832\" (UID: \"46307ab4-5e52-4d6d-a527-9e7756732832\") "
Oct 31 02:19:44.823889 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.823149    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-ca-configmap" pod "46307ab4-5e52-4d6d-a527-9e7756732832" (UID: "46307ab4-5e52-4d6d-a527-9e7756732832"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:19:44.825509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.825449    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-kubeconfig" pod "46307ab4-5e52-4d6d-a527-9e7756732832" (UID: "46307ab4-5e52-4d6d-a527-9e7756732832"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:19:44.825509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.825477    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-kube-api-access-dlw2h" pod "46307ab4-5e52-4d6d-a527-9e7756732832" (UID: "46307ab4-5e52-4d6d-a527-9e7756732832"). InnerVolumeSpecName "kube-api-access-dlw2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:44.825679 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.825546    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-cacerts" pod "46307ab4-5e52-4d6d-a527-9e7756732832" (UID: "46307ab4-5e52-4d6d-a527-9e7756732832"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:19:44.825876 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.825837    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-istio-token" pod "46307ab4-5e52-4d6d-a527-9e7756732832" (UID: "46307ab4-5e52-4d6d-a527-9e7756732832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:44.827011 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.826986    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-dns-cert" pod "46307ab4-5e52-4d6d-a527-9e7756732832" (UID: "46307ab4-5e52-4d6d-a527-9e7756732832"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:19:44.832397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.832374    5293 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46307ab4-5e52-4d6d-a527-9e7756732832-local-certs" pod "46307ab4-5e52-4d6d-a527-9e7756732832" (UID: "46307ab4-5e52-4d6d-a527-9e7756732832"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:44.923660 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.923631    5293 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-cacerts\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:44.923660 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.923656    5293 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlw2h\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-kube-api-access-dlw2h\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:44.923660 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.923667    5293 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-ca-configmap\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:44.923897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.923676    5293 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46307ab4-5e52-4d6d-a527-9e7756732832-local-certs\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:44.923897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.923685    5293 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-csr-dns-cert\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:44.923897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.923693    5293 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46307ab4-5e52-4d6d-a527-9e7756732832-istio-kubeconfig\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:44.923897 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:44.923702    5293 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46307ab4-5e52-4d6d-a527-9e7756732832-istio-token\") on node \"ip-10-0-130-49.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:45.165742 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.165704    5293 generic.go:356] "Generic (PLEG): container finished" podID="46307ab4-5e52-4d6d-a527-9e7756732832" containerID="9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d" exitCode=0
Oct 31 02:19:45.165939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.165748    5293 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"
Oct 31 02:19:45.165939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.165785    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm" event={"ID":"46307ab4-5e52-4d6d-a527-9e7756732832","Type":"ContainerDied","Data":"9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d"}
Oct 31 02:19:45.165939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.165842    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm" event={"ID":"46307ab4-5e52-4d6d-a527-9e7756732832","Type":"ContainerDied","Data":"ca66fb40770478a5403f931d0eb791ce64d7861da8a81c92dda02011a8a99bca"}
Oct 31 02:19:45.165939 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.165862    5293 scope.go:122] "RemoveContainer" containerID="9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d"
Oct 31 02:19:45.181831 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.181811    5293 scope.go:122] "RemoveContainer" containerID="9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d"
Oct 31 02:19:45.182212 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:19:45.182189    5293 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d\": container with ID starting with 9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d not found: ID does not exist" containerID="9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d"
Oct 31 02:19:45.182307 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.182220    5293 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d"} err="failed to get container status \"9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d\": rpc error: code = NotFound desc = could not find container \"9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d\": container with ID starting with 9cf97a196edd13cbdb87561f9efc3bdada9cd55e8c33d4aa16cd652a89401a0d not found: ID does not exist"
Oct 31 02:19:45.187850 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.187830    5293 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"]
Oct 31 02:19:45.192857 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:45.192833    5293 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-chwsm"]
Oct 31 02:19:46.164165 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:46.164131    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sgnh6_552bfc4c-f3a9-4b18-a7d2-4f7e3d8aa2d6/global-pull-secret-syncer/0.log"
Oct 31 02:19:46.174685 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:46.174644    5293 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46307ab4-5e52-4d6d-a527-9e7756732832" path="/var/lib/kubelet/pods/46307ab4-5e52-4d6d-a527-9e7756732832/volumes"
Oct 31 02:19:46.385524 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:46.385478    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c9rzr_fb0eaa17-d3d8-4db5-b63c-1d74a7d77996/konnectivity-agent/0.log"
Oct 31 02:19:46.477710 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:46.477621    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-49.us-east-2.compute.internal_b3e5d96baa84771b6810c7d4e7a868fe/haproxy/0.log"
Oct 31 02:19:51.091438 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:51.091408    5293 scope.go:122] "RemoveContainer" containerID="9dd82c039820d0f4e5f1bd6e5893e526a53085d27d46f662eba0a436cc49c044"
Oct 31 02:19:51.102430 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:51.102410    5293 scope.go:122] "RemoveContainer" containerID="a65cb507b1adfefd6a46decf4b7b71e78ca857629c441cac281e450815f4ba11"
Oct 31 02:19:51.113916 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:51.113891    5293 scope.go:122] "RemoveContainer" containerID="c68c058770487a222ba212b825ab9b08eee8e64e84b72fc0b829c0c9f502bf6c"
Oct 31 02:19:51.123054 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:51.123016    5293 scope.go:122] "RemoveContainer" containerID="15667c201cc0c23d5f7857f61a4ffff8c5e2ff3ba2a010bdb39b762f22d0ec45"
Oct 31 02:19:51.131606 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:51.131588    5293 scope.go:122] "RemoveContainer" containerID="b1f1ef8f170c6b767e53e731e32bc4f99d15fbf8a39c62ec81375e1c87ca91aa"
Oct 31 02:19:51.139591 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:51.139569    5293 scope.go:122] "RemoveContainer" containerID="3114a6f9b64a35f63d877bfe30c44d046d3ee00640bea4c372f09f541f1a5743"
Oct 31 02:19:53.150292 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:53.150261    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8wgfx_828a21ed-5fd5-40d8-9a6f-d2701ab2f5ba/kuadrant-console-plugin/0.log"
Oct 31 02:19:53.624634 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:53.624542    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-46qd9_771a7056-0944-443b-969c-6d4283bec9bf/manager/0.log"
Oct 31 02:19:53.663160 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:53.663118    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-786vs_d61f77a9-2b32-463f-bc28-64e21cd43a7b/limitador/0.log"
Oct 31 02:19:55.240436 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.240403    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfpsd/must-gather-9dvk9" event={"ID":"250a107d-9b15-4ac7-8168-b0d40e29b013","Type":"ContainerStarted","Data":"0920d9e34d6dc0ac8e42fc668779511152422218cfa153329b9e6fb43be2f8a8"}
Oct 31 02:19:55.296853 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.296788    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hxnfz_d11ae2b3-41ce-4c4f-9e94-0fb9330700be/kube-state-metrics/0.log"
Oct 31 02:19:55.326295 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.326254    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hxnfz_d11ae2b3-41ce-4c4f-9e94-0fb9330700be/kube-rbac-proxy-main/0.log"
Oct 31 02:19:55.358120 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.358088    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hxnfz_d11ae2b3-41ce-4c4f-9e94-0fb9330700be/kube-rbac-proxy-self/0.log"
Oct 31 02:19:55.389531 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.389502    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f555ffffc-q4tx9_e435a839-2e87-46cf-a521-17e5ffa2dfc0/metrics-server/0.log"
Oct 31 02:19:55.422847 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.422816    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-mgfcp_a0e1fa2a-b9a6-479e-a0ab-ad118dd4c47b/monitoring-plugin/0.log"
Oct 31 02:19:55.459456 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.459429    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8cc5t_88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/node-exporter/0.log"
Oct 31 02:19:55.486440 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.486413    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8cc5t_88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/kube-rbac-proxy/0.log"
Oct 31 02:19:55.512945 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.512906    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8cc5t_88d6a153-63a8-4b4a-9ca4-e3f1796eb34b/init-textfile/0.log"
Oct 31 02:19:55.838509 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.838474    5293 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"]
Oct 31 02:19:55.839111 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.839091    5293 cpu_manager.go:414] "containerMap: removing container" podUID="46307ab4-5e52-4d6d-a527-9e7756732832" containerName="discovery"
Oct 31 02:19:55.839206 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.839113    5293 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46307ab4-5e52-4d6d-a527-9e7756732832" containerName="discovery"
Oct 31 02:19:55.839279 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.839233    5293 memory_manager.go:365] "RemoveStaleState removing state" podUID="46307ab4-5e52-4d6d-a527-9e7756732832" containerName="discovery"
Oct 31 02:19:55.845234 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.845210    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:55.856441 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.856417    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"]
Oct 31 02:19:55.856736 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.856717    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kprhq\"/\"openshift-service-ca.crt\""
Oct 31 02:19:55.856879 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.856862    5293 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kprhq\"/\"default-dockercfg-2jdb2\""
Oct 31 02:19:55.856980 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.856965    5293 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kprhq\"/\"kube-root-ca.crt\""
Oct 31 02:19:55.903906 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.903878    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-97pjd_dd5f332c-3b57-4a47-9dae-f191abaa5d52/kube-rbac-proxy-main/0.log"
Oct 31 02:19:55.931933 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.931888    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-97pjd_dd5f332c-3b57-4a47-9dae-f191abaa5d52/kube-rbac-proxy-self/0.log"
Oct 31 02:19:55.940915 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.940885    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-proc\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:55.941098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.940985    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdgg\" (UniqueName: \"kubernetes.io/projected/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-kube-api-access-gsdgg\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:55.941098 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.941025    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-sys\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:55.941201 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.941180    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-lib-modules\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:55.941245 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.941229    5293 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-podres\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:55.967006 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:55.966979    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-97pjd_dd5f332c-3b57-4a47-9dae-f191abaa5d52/openshift-state-metrics/0.log"
Oct 31 02:19:56.042660 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.042584    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdgg\" (UniqueName: \"kubernetes.io/projected/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-kube-api-access-gsdgg\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.042854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.042718    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-sys\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.042854 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.042819    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-lib-modules\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.042983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.042873    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-sys\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.042983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.042947    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-podres\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.042983 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.042958    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-lib-modules\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.043173 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.043132    5293 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-proc\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.043232 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.043134    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-podres\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.043232 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.043210    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-proc\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.054235 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.054189    5293 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdgg\" (UniqueName: \"kubernetes.io/projected/03fd19f2-9f52-4ebd-87ad-72871fa35d3f-kube-api-access-gsdgg\") pod \"perf-node-gather-daemonset-7sl29\" (UID: \"03fd19f2-9f52-4ebd-87ad-72871fa35d3f\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.169325 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.169172    5293 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:56.179667 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.179642    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s9rxb_e75104bf-ae55-488f-bab5-595defac1e17/server/0.log"
Oct 31 02:19:56.254849 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.254817    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfpsd/must-gather-9dvk9" event={"ID":"250a107d-9b15-4ac7-8168-b0d40e29b013","Type":"ContainerStarted","Data":"9613e6c4e51ad2b642dfe22b515e3bc75e0143e5fad1f8941545250020509c95"}
Oct 31 02:19:56.276789 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.275202    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zfpsd/must-gather-9dvk9" podStartSLOduration=2.142155842 podStartE2EDuration="14.275179776s" totalImagesPullingTime="12.133023934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:19:56.272010473 +0000 UTC m=+3554.220658890" watchObservedRunningTime="2026-10-31 02:19:56.275179776 +0000 UTC m=+3554.223828168"
Oct 31 02:19:56.340774 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.340740    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5754dc898c-kx4q9_08d155b0-ae14-4540-acf9-070068acc05a/telemeter-client/0.log"
Oct 31 02:19:56.374768 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.374739    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5754dc898c-kx4q9_08d155b0-ae14-4540-acf9-070068acc05a/reload/0.log"
Oct 31 02:19:56.404276 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.404235    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5754dc898c-kx4q9_08d155b0-ae14-4540-acf9-070068acc05a/kube-rbac-proxy/0.log"
Oct 31 02:19:56.473521 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:56.473492    5293 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"]
Oct 31 02:19:56.476688 ip-10-0-130-49 kubenswrapper[5293]: W1031 02:19:56.476633    5293 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod03fd19f2_9f52_4ebd_87ad_72871fa35d3f.slice/crio-e8b2b470cd4ffcc02d13010f7fd8746751112b6982d6686f3f6d0246e739f5de WatchSource:0}: Error finding container e8b2b470cd4ffcc02d13010f7fd8746751112b6982d6686f3f6d0246e739f5de: Status 404 returned error can't find the container with id e8b2b470cd4ffcc02d13010f7fd8746751112b6982d6686f3f6d0246e739f5de
Oct 31 02:19:57.027258 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:57.027222    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4bjhg_ba3ae4d1-4d61-457c-a552-b4001c770117/manager/0.log"
Oct 31 02:19:57.265317 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:57.265261    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29" event={"ID":"03fd19f2-9f52-4ebd-87ad-72871fa35d3f","Type":"ContainerStarted","Data":"e5fbeb00946be9f306f536cba0cec57797054af330704d338972f37749dbd7d7"}
Oct 31 02:19:57.265838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:57.265330    5293 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29" event={"ID":"03fd19f2-9f52-4ebd-87ad-72871fa35d3f","Type":"ContainerStarted","Data":"e8b2b470cd4ffcc02d13010f7fd8746751112b6982d6686f3f6d0246e739f5de"}
Oct 31 02:19:57.265838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:57.265672    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:19:57.284593 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:57.284484    5293 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29" podStartSLOduration=2.284465679 podStartE2EDuration="2.284465679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:19:57.282452731 +0000 UTC m=+3555.231101124" watchObservedRunningTime="2026-10-31 02:19:57.284465679 +0000 UTC m=+3555.233114072"
Oct 31 02:19:57.859710 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:57.859678    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s9rxb_e75104bf-ae55-488f-bab5-595defac1e17/server/0.log"
Oct 31 02:19:58.801649 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:58.801614    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4bjhg_ba3ae4d1-4d61-457c-a552-b4001c770117/manager/0.log"
Oct 31 02:19:59.217555 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:59.217530    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s9rxb_e75104bf-ae55-488f-bab5-595defac1e17/server/0.log"
Oct 31 02:19:59.941827 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:19:59.941766    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4bjhg_ba3ae4d1-4d61-457c-a552-b4001c770117/manager/0.log"
Oct 31 02:20:01.125856 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:01.125828    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s9rxb_e75104bf-ae55-488f-bab5-595defac1e17/server/0.log"
Oct 31 02:20:01.647713 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:01.647670    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jtnln_a3cb3e38-466f-4fab-8281-1257ca8fc8ae/dns/0.log"
Oct 31 02:20:01.678296 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:01.678227    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jtnln_a3cb3e38-466f-4fab-8281-1257ca8fc8ae/kube-rbac-proxy/0.log"
Oct 31 02:20:01.843671 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:01.843576    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jp6vv_4950850b-9fc3-4c93-a3c8-b92ef5d906cc/dns-node-resolver/0.log"
Oct 31 02:20:02.117010 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:02.116932    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4bjhg_ba3ae4d1-4d61-457c-a552-b4001c770117/manager/0.log"
Oct 31 02:20:02.218502 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:02.218477    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s9rxb_e75104bf-ae55-488f-bab5-595defac1e17/server/0.log"
Oct 31 02:20:02.612569 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:02.612538    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mg8qc_471cd70e-c62e-4f48-903d-1d71fa617d35/node-ca/0.log"
Oct 31 02:20:02.914397 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:02.914356    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4bjhg_ba3ae4d1-4d61-457c-a552-b4001c770117/manager/0.log"
Oct 31 02:20:03.284274 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:03.284208    5293 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-7sl29"
Oct 31 02:20:04.445733 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:04.445702    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s9rxb_e75104bf-ae55-488f-bab5-595defac1e17/server/0.log"
Oct 31 02:20:05.814272 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:05.814236    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4bjhg_ba3ae4d1-4d61-457c-a552-b4001c770117/manager/0.log"
Oct 31 02:20:06.924838 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:06.924800    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5p96n_1b6a6c52-4b49-4647-b86f-b2a91537032c/discovery/0.log"
Oct 31 02:20:07.131472 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:07.131434    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5p96n_1b6a6c52-4b49-4647-b86f-b2a91537032c/discovery/0.log"
Oct 31 02:20:09.622144 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:09.622105    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5p96n_1b6a6c52-4b49-4647-b86f-b2a91537032c/discovery/0.log"
Oct 31 02:20:09.890576 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:09.890505    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5p96n_1b6a6c52-4b49-4647-b86f-b2a91537032c/discovery/0.log"
Oct 31 02:20:11.777613 ip-10-0-130-49 kubenswrapper[5293]: E1031 02:20:11.777576    5293 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs2gb" containerName="extractor"
Oct 31 02:20:12.788746 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:12.788706    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b44bd95d7-tdrn5_d6956b96-691a-445b-89e2-2fe5b289674e/kube-auth-proxy/0.log"
Oct 31 02:20:12.837379 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:12.837349    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-6dz48_774c137c-c103-4d59-b525-37ad6df863e5/istio-proxy/0.log"
Oct 31 02:20:12.911755 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:12.911725    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-d8xb2_0cef45b4-73c6-463c-a1e0-716629cc73b0/istio-proxy/0.log"
Oct 31 02:20:12.991988 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:12.991961    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-v46nb_d67d9b4f-a0df-43a7-97b7-88df20e0aa19/istio-proxy/0.log"
Oct 31 02:20:13.016237 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:13.016206    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-x47hr_a36f3ee8-e967-4465-a485-7a9b040e02c2/istio-proxy/0.log"
Oct 31 02:20:13.130304 ip-10-0-130-49 kubenswrapper[5293]: I1031 02:20:13.130239    5293 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-cxqrw_da6a4c1a-a546-4493-8003-6e4fbc1e84a3/payload-pre-processing/0.log"